kristoferssolo/telescope-frecency.nvim
1
0
Fork 0
mirror of https://github.com/kristoferssolo/telescope-frecency.nvim.git synced 2025-10-21 20:10:38 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

feat!: remove code for SQLite (#172)

Browse Source 
* feat!: remove code for SQLite

ref [Introduce revised telescope-frecency.nvim : neovim](https://www.reddit.com/r/neovim/comments/174m8zu/introduce_revised_telescopefrecencynvim/)

I have deprecated SQLite features 4 months ago. It is the time to remove
code for them.

* test: fix test to load telescope validly

* test: remove sqlite.lua from CI settings

* test: test database as native

* fix: add lacked type from old database/sqlite.lua

* docs: remove description for SQLite3 logic

* chore: fix types

* chore: add types for Database:raw_table
This commit is contained in:
JINNOUCHI Yasushi 2024-01-30 18:26:07 +09:00 committed by GitHub
parent a3e818d001
commit ada91ca486
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
16 changed files with 527 additions and 1101 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.github/workflows/ci.yml vendored
View File

@ -30,11 +30,6 @@ jobs:
with:
repository: nvim-telescope/telescope.nvim
path: telescope.nvim
- name: Checkout sqlite.lua
uses: actions/checkout@v3
with:
repository: kkharji/sqlite.lua
path: sqlite.lua
- name: Install Neovim
uses: rhysd/action-setup-vim@v1
id: nvim
@ -45,7 +40,6 @@ jobs:
env:
PLENARY_PATH: plenary.nvim
TELESCOPE_PATH: telescope.nvim
SQLITE_PATH: sqlite.lua
DEBUG_PLENARY: 1
EXE: ${{ steps.nvim.outputs.executable }}
run: |-
@ -59,14 +53,12 @@ jobs:
env:
PLENARY_PATH: plenary.nvim
TELESCOPE_PATH: telescope.nvim
SQLITE_PATH: sqlite.lua
DEBUG_PLENARY: 1
EXE: ${{ steps.nvim.outputs.executable }}
run: |-
# HACK: This is needed because it fails to add runtimepath's.
cp -af $PLENARY_PATH/lua/plenary/ lua/
cp -af $TELESCOPE_PATH/lua/telescope/ lua/
cp -af $SQLITE_PATH/lua/sqlite/ lua/
TEST_DIR=lua/frecency/tests/
MINIMAL_LUA=${TEST_DIR}minimal.lua
NVIM=$(perl -e '$_ = $ENV{EXE}; s,\\,/,g; print')

41
README.md
View File

@ -81,20 +81,11 @@ directories provided by the language server.
- [nvim-web-devicons](https://github.com/kyazdani42/nvim-web-devicons) (optional)
- [ripgrep](https://github.com/BurntSushi/ripgrep) or [fd](https://github.com/sharkdp/fd) (optional)
**NOTE:** The former version of this plugin has used [SQLite3][] database to
store timestamps and file records. But the current build uses Lua native code
to store them, so you can now remove [sqlite.lua][] from dependencies. See
[*Remove dependency for sqlite.lua*][remove-sqlite] for the detail.
**NOTE:** `ripgrep` or `fd` will be used to list up workspace files. They are
extremely faster than the native Lua logic. If you don't have them, it
fallbacks to Lua code automatically. See the detail for `workspace_scan_cmd`
option.
[SQLite3]: https://www.sqlite.org/index.html
[sqlite.lua]: https://github.com/kkharji/sqlite.lua
[remove-sqlite]: #user-content-remove-dependency-for-sqlitelua
## Installation
### [Packer.nvim](https://github.com/wbthomason/packer.nvim)
@ -224,11 +215,6 @@ See [default configuration](https://github.com/nvim-telescope/telescope.nvim#tel
Determines if non-indexed files are included in workspace filter results.
- `use_sqlite` (default: `false`)
Use [sqlite.lua][] with `true` or native code with `false`. See [*Remove
dependency for sqlite.lua*][remove-sqlite] for the detail.
- `workspace_scan_cmd` (default: `nil`)
This option can be set values as `"LUA"|string[]|nil`. With the default
@ -280,7 +266,7 @@ telescope.setup {
The default location for the database is `$XDG_DATA_HOME/nvim` (eg
`~/.local/share/nvim/` on linux). This can be configured with the `db_root`
config option.
config option. The filename for the database is `file_frecency.bin`.
### Maintainance
@ -318,21 +304,22 @@ not remove the file itself, only from DB.
:FrecencyDelete /full/path/to/the/file
```
### Remove dependency for [sqlite.lua][]
### Note about the compatibility for the former version.
The former version of this plugin has used SQLite3 library to store data. When
you upgrade from such version, Neovim will silently migrate DB and inform that
you can remove `sqlite.lua` from dependencies.
The former version of this plugin has used SQLite3 library to store data. [#172][]
has removed the whole code for that. If you prefer the old SQLite database,
you should lock the version to [a3e818d][] with your favorite plugin manager.
| made by default | made by `sqlite.lua` |
|--|--|
| `~/.local/share/nvim/file_frecency.bin` | `~/.local/share/nvim/file_frecency.sqlite3` |
[#172]: https://github.com/nvim-telescope/telescope-frecency.nvim/pull/172
[a3e818d]: https://github.com/nvim-telescope/telescope-frecency.nvim/commit/a3e818d001baad9ee2f6800d3bbc71c4275364ae
The DB file will be migrated into a filename above, and old file (SQLite3
version) will still remain. If you still want to use SQLite3 version, set
`use_sqlite = true`.
Also you can explicitly migrate DB by calling `:FrecencyMigrateDB` command.
```lua
-- example for lazy.nvim
{
"nvim-telescope/telescope-frecency.nvim",
commit = "a3e818d001baad9ee2f6800d3bbc71c4275364ae",
}
```
## Highlight Groups

247
lua/frecency/database.lua
View File

@ -1,4 +1,10 @@
---@diagnostic disable: missing-return, unused-local
local FileLock = require "frecency.file_lock"
local wait = require "frecency.wait"
local watcher = require "frecency.watcher"
local log = require "plenary.log"
local async = require "plenary.async" --[[@as PlenaryAsync]]
local Path = require "plenary.path" --[[@as PlenaryPath]]
---@class FrecencyDatabaseConfig
---@field root string
@ -6,43 +12,220 @@
---@field path string?
---@field workspace string?
---@class FrecencyDatabase
---@field config FrecencyDatabaseConfig
---@field filename string
---@field has_entry fun(): boolean
---@field new fun(fs: FrecencyFS, config: FrecencyDatabaseConfig): FrecencyDatabase
---@field protected fs FrecencyFS
local Database = {}
---@param paths string[]
---@return nil
function Database:insert_files(paths) end
---@return integer[]|string[]
function Database:unlinked_entries() end
---@param files integer[]|string[]
---@return nil
function Database:remove_files(files) end
---@param path string
---@return boolean
function Database:remove_entry(path) end
---@param path string
---@param max_count integer
---@param datetime string?
---@return nil
function Database:update(path, max_count, datetime) end
---@async
---@class FrecencyDatabaseEntry
---@field ages number[]
---@field count integer
---@field path string
---@field score number
---@class FrecencyDatabase
---@field config FrecencyDatabaseConfig
---@field file_lock FrecencyFileLock
---@field filename string
---@field fs FrecencyFS
---@field new fun(fs: FrecencyFS, config: FrecencyDatabaseConfig): FrecencyDatabase
---@field table FrecencyDatabaseTable
---@field version "v1"
local Database = {}
---@class FrecencyDatabaseTable
---@field version string
---@field records table<string,FrecencyDatabaseRecord>
---@class FrecencyDatabaseRecord
---@field count integer
---@field timestamps integer[]
---@param fs FrecencyFS
---@param config FrecencyDatabaseConfig
---@return FrecencyDatabase
Database.new = function(fs, config)
local version = "v1"
local self = setmetatable({
config = config,
fs = fs,
table = { version = version, records = {} },
version = version,
}, { __index = Database })
self.filename = Path.new(self.config.root, "file_frecency.bin").filename
self.file_lock = FileLock.new(self.filename)
local tx, rx = async.control.channel.counter()
watcher.watch(self.filename, tx)
wait(function()
self:load()
end)
async.void(function()
while true do
rx.last()
log.debug "file changed. loading..."
self:load()
end
end)()
return self
end
---@return boolean
function Database:has_entry()
return not vim.tbl_isempty(self.table.records)
end
---@param paths string[]
---@return nil
function Database:insert_files(paths)
if #paths == 0 then
return
end
for _, path in ipairs(paths) do
self.table.records[path] = { count = 1, timestamps = { 0 } }
end
wait(function()
self:save()
end)
end
---@return string[]
function Database:unlinked_entries()
local paths = {}
for file in pairs(self.table.records) do
if not self.fs:is_valid_path(file) then
table.insert(paths, file)
end
end
return paths
end
---@param paths string[]
function Database:remove_files(paths)
for _, file in ipairs(paths) do
self.table.records[file] = nil
end
wait(function()
self:save()
end)
end
---@param path string
---@param max_count integer
---@param datetime string?
function Database:update(path, max_count, datetime)
local record = self.table.records[path] or { count = 0, timestamps = {} }
record.count = record.count + 1
local now = self:now(datetime)
table.insert(record.timestamps, now)
if #record.timestamps > max_count then
local new_table = {}
for i = #record.timestamps - max_count + 1, #record.timestamps do
table.insert(new_table, record.timestamps[i])
end
record.timestamps = new_table
end
self.table.records[path] = record
wait(function()
self:save()
end)
end
---@param workspace string?
---@param datetime string?
---@return FrecencyDatabaseEntry[]
function Database:get_entries(workspace, datetime) end
function Database:get_entries(workspace, datetime)
local now = self:now(datetime)
local items = {}
for path, record in pairs(self.table.records) do
if self.fs:starts_with(path, workspace) then
table.insert(items, {
path = path,
count = record.count,
ages = vim.tbl_map(function(v)
return (now - v) / 60
end, record.timestamps),
})
end
end
return items
end
-- TODO: remove this func
-- This is a func for testing
---@private
---@param datetime string?
---@return integer
function Database:now(datetime)
if not datetime then
return os.time()
end
local epoch
wait(function()
local tz_fix = datetime:gsub("+(%d%d):(%d%d)$", "+%1%2")
epoch = require("frecency.tests.util").time_piece(tz_fix)
end)
return epoch
end
---@async
---@return nil
function Database:load()
local start = os.clock()
local err, data = self.file_lock:with(function()
local err, stat = async.uv.fs_stat(self.filename)
if err then
return nil
end
local fd
err, fd = async.uv.fs_open(self.filename, "r", tonumber("644", 8))
assert(not err, err)
local data
err, data = async.uv.fs_read(fd, stat.size)
assert(not err, err)
assert(not async.uv.fs_close(fd))
watcher.update(stat)
return data
end)
assert(not err, err)
local tbl = loadstring(data or "")() --[[@as FrecencyDatabaseTable?]]
if tbl and tbl.version == self.version then
self.table = tbl
end
log.debug(("load() takes %f seconds"):format(os.clock() - start))
end
---@async
---@return nil
function Database:save()
local start = os.clock()
local err = self.file_lock:with(function()
self:raw_save(self.table)
local err, stat = async.uv.fs_stat(self.filename)
assert(not err, err)
watcher.update(stat)
return nil
end)
assert(not err, err)
log.debug(("save() takes %f seconds"):format(os.clock() - start))
end
---@async
---@param tbl FrecencyDatabaseTable
function Database:raw_save(tbl)
local f = assert(load("return " .. vim.inspect(tbl)))
local data = string.dump(f)
local err, fd = async.uv.fs_open(self.filename, "w", tonumber("644", 8))
assert(not err, err)
assert(not async.uv.fs_write(fd, data))
assert(not async.uv.fs_close(fd))
end
---@param path string
---@return boolean
function Database:remove_entry(path)
if not self.table.records[path] then
return false
end
self.table.records[path] = nil
wait(function()
self:save()
end)
return true
end
return Database

212
lua/frecency/database/native.lua
View File

@ -1,212 +0,0 @@
local FileLock = require "frecency.file_lock"
local wait = require "frecency.wait"
local watcher = require "frecency.database.native.watcher"
local log = require "plenary.log"
local async = require "plenary.async" --[[@as PlenaryAsync]]
local Path = require "plenary.path" --[[@as PlenaryPath]]
---@class FrecencyDatabaseNative: FrecencyDatabase
---@field version "v1"
---@field file_lock FrecencyFileLock
---@field table FrecencyDatabaseNativeTable
local Native = {}
---@class FrecencyDatabaseNativeTable
---@field version string
---@field records table<string,FrecencyDatabaseNativeRecord>
---@class FrecencyDatabaseNativeRecord
---@field count integer
---@field timestamps integer[]
---@param fs FrecencyFS
---@param config FrecencyDatabaseConfig
---@return FrecencyDatabaseNative
Native.new = function(fs, config)
local version = "v1"
local self = setmetatable({
config = config,
fs = fs,
table = { version = version, records = {} },
version = version,
}, { __index = Native })
self.filename = Path.new(self.config.root, "file_frecency.bin").filename
self.file_lock = FileLock.new(self.filename)
local tx, rx = async.control.channel.counter()
watcher.watch(self.filename, tx)
wait(function()
self:load()
end)
async.void(function()
while true do
rx.last()
log.debug "file changed. loading..."
self:load()
end
end)()
return self
end
---@return boolean
function Native:has_entry()
return not vim.tbl_isempty(self.table.records)
end
---@param paths string[]
---@return nil
function Native:insert_files(paths)
if #paths == 0 then
return
end
for _, path in ipairs(paths) do
self.table.records[path] = { count = 1, timestamps = { 0 } }
end
wait(function()
self:save()
end)
end
---@return string[]
function Native:unlinked_entries()
local paths = {}
for file in pairs(self.table.records) do
if not self.fs:is_valid_path(file) then
table.insert(paths, file)
end
end
return paths
end
---@param paths string[]
function Native:remove_files(paths)
for _, file in ipairs(paths) do
self.table.records[file] = nil
end
wait(function()
self:save()
end)
end
---@param path string
---@param max_count integer
---@param datetime string?
function Native:update(path, max_count, datetime)
local record = self.table.records[path] or { count = 0, timestamps = {} }
record.count = record.count + 1
local now = self:now(datetime)
table.insert(record.timestamps, now)
if #record.timestamps > max_count then
local new_table = {}
for i = #record.timestamps - max_count + 1, #record.timestamps do
table.insert(new_table, record.timestamps[i])
end
record.timestamps = new_table
end
self.table.records[path] = record
wait(function()
self:save()
end)
end
---@param workspace string?
---@param datetime string?
---@return FrecencyDatabaseEntry[]
function Native:get_entries(workspace, datetime)
local now = self:now(datetime)
local items = {}
for path, record in pairs(self.table.records) do
if self.fs:starts_with(path, workspace) then
table.insert(items, {
path = path,
count = record.count,
ages = vim.tbl_map(function(v)
return (now - v) / 60
end, record.timestamps),
})
end
end
return items
end
-- TODO: remove this func
-- This is a func for testing
---@private
---@param datetime string?
---@return integer
function Native:now(datetime)
if not datetime then
return os.time()
end
local epoch
wait(function()
local tz_fix = datetime:gsub("+(%d%d):(%d%d)$", "+%1%2")
epoch = require("frecency.tests.util").time_piece(tz_fix)
end)
return epoch
end
---@async
---@return nil
function Native:load()
local start = os.clock()
local err, data = self.file_lock:with(function()
local err, stat = async.uv.fs_stat(self.filename)
if err then
return nil
end
local fd
err, fd = async.uv.fs_open(self.filename, "r", tonumber("644", 8))
assert(not err, err)
local data
err, data = async.uv.fs_read(fd, stat.size)
assert(not err, err)
assert(not async.uv.fs_close(fd))
watcher.update(stat)
return data
end)
assert(not err, err)
local tbl = loadstring(data or "")() --[[@as FrecencyDatabaseNativeTable?]]
if tbl and tbl.version == self.version then
self.table = tbl
end
log.debug(("load() takes %f seconds"):format(os.clock() - start))
end
---@async
---@return nil
function Native:save()
local start = os.clock()
local err = self.file_lock:with(function()
self:raw_save(self.table)
local err, stat = async.uv.fs_stat(self.filename)
assert(not err, err)
watcher.update(stat)
return nil
end)
assert(not err, err)
log.debug(("save() takes %f seconds"):format(os.clock() - start))
end
function Native:raw_save(tbl)
local f = assert(load("return " .. vim.inspect(tbl)))
local data = string.dump(f)
local err, fd = async.uv.fs_open(self.filename, "w", tonumber("644", 8))
assert(not err, err)
assert(not async.uv.fs_write(fd, data))
assert(not async.uv.fs_close(fd))
end
---@param path string
---@return boolean
function Native:remove_entry(path)
if not self.table.records[path] then
return false
end
self.table.records[path] = nil
wait(function()
self:save()
end)
return true
end
return Native

155
lua/frecency/database/sqlite.lua
View File

@ -1,155 +0,0 @@
local sqlite = require "frecency.sqlite"
local log = require "plenary.log"
local Path = require "plenary.path" --[[@as PlenaryPath]]
---@class FrecencySqliteDB: sqlite_db
---@field files sqlite_tbl
---@field timestamps sqlite_tbl
---@class FrecencyFile
---@field count integer
---@field id integer
---@field path string
---@field score integer calculated from count and age
---@class FrecencyTimestamp
---@field age integer calculated from timestamp
---@field file_id integer
---@field id integer
---@field timestamp number
---@class FrecencyDatabaseSqlite: FrecencyDatabase
---@field sqlite FrecencySqliteDB
local Sqlite = {}
---@param fs FrecencyFS
---@param config FrecencyDatabaseConfig
---@return FrecencyDatabaseSqlite
Sqlite.new = function(fs, config)
local self = setmetatable(
{ config = config, buf_registered_flag_name = "telescope_frecency_registered", fs = fs },
{ __index = Sqlite }
)
self.filename = Path.new(self.config.root, "file_frecency.sqlite3").filename
self.sqlite = setmetatable({}, {
__index = function(this, key)
if not rawget(this, "instance") then
local lib = sqlite.lib
rawset(
this,
"instance",
sqlite {
uri = self.filename,
files = { id = true, count = { "integer", default = 1, required = true }, path = "string" },
timestamps = {
id = true,
file_id = { "integer", reference = "files.id", on_delete = "cascade" },
timestamp = { "real", default = lib.julianday "now" },
},
}
)
end
return rawget(this, "instance")[key]
end,
})
return self
end
---@return boolean
function Sqlite:has_entry()
return self.sqlite.files:count() > 0
end
---@param paths string[]
---@return integer
function Sqlite:insert_files(paths)
if #paths == 0 then
return 0
end
---@param path string
return self.sqlite.files:insert(vim.tbl_map(function(path)
return { path = path, count = 0 } -- TODO: remove when sql.nvim#97 is closed
end, paths))
end
---@param workspace string?
---@param datetime string?
---@return FrecencyDatabaseEntry[]
function Sqlite:get_entries(workspace, datetime)
local query = workspace and { contains = { path = { workspace .. "/*" } } } or {}
log.debug { query = query }
local files = self.sqlite.files:get(query) --[[@as FrecencyFile[] ]]
local lib = sqlite.lib
local age = lib.cast((lib.julianday(datetime) - lib.julianday "timestamp") * 24 * 60, "integer")
local timestamps = self.sqlite.timestamps:get { keys = { age = age, "id", "file_id" } } --[[@as FrecencyTimestamp[] ]]
---@type table<integer,number[]>
local age_map = {}
for _, timestamp in ipairs(timestamps) do
if not age_map[timestamp.file_id] then
age_map[timestamp.file_id] = {}
end
table.insert(age_map[timestamp.file_id], timestamp.age)
end
local items = {}
for _, file in ipairs(files) do
table.insert(items, { path = file.path, count = file.count, ages = age_map[file.id] })
end
return items
end
---@param datetime string? ISO8601 format string
---@return FrecencyTimestamp[]
function Sqlite:get_timestamps(datetime)
local lib = sqlite.lib
local age = lib.cast((lib.julianday(datetime) - lib.julianday "timestamp") * 24 * 60, "integer")
return self.sqlite.timestamps:get { keys = { age = age, "id", "file_id" } }
end
---@param path string
---@param count integer
---@param datetime string?
---@return nil
function Sqlite:update(path, count, datetime)
local file = self.sqlite.files:get({ where = { path = path } })[1] --[[@as FrecencyFile?]]
local file_id
if file then
self.sqlite.files:update { where = { id = file.id }, set = { count = file.count + 1 } }
file_id = file.id
else
file_id = self.sqlite.files:insert { path = path }
end
self.sqlite.timestamps:insert {
file_id = file_id,
timestamp = datetime and sqlite.lib.julianday(datetime) or nil,
}
local timestamps = self.sqlite.timestamps:get { where = { file_id = file_id } } --[[@as FrecencyTimestamp[] ]]
local trim_at = timestamps[#timestamps - count + 1]
if trim_at then
self.sqlite.timestamps:remove { file_id = tostring(file_id), id = "<" .. tostring(trim_at.id) }
end
end
---@return integer[]
function Sqlite:unlinked_entries()
---@param file FrecencyFile
return self.sqlite.files:map(function(file)
if not self.fs:is_valid_path(file.path) then
return file.id
end
end)
end
---@param ids integer[]
---@return nil
function Sqlite:remove_files(ids)
self.sqlite.files:remove { id = ids }
end
---@param path string
---@return boolean
function Sqlite:remove_entry(path)
local exists = not not self.sqlite.files:get({ where = { path = path } })[1]
return exists and self.sqlite.files:remove { path = path } or false
end
return Sqlite

6
lua/frecency/entry_maker.lua
View File

@ -40,6 +40,12 @@ end
---@field score number
---@field display fun(entry: FrecencyEntry): string, table
---@class FrecencyFile
---@field count integer
---@field id integer
---@field path string
---@field score integer calculated from count and age
---@alias FrecencyEntryMakerInstance fun(file: FrecencyFile): FrecencyEntry
---@param filepath_formatter FrecencyFilepathFormatter

73
lua/frecency/frecency.lua
View File

@ -1,12 +1,9 @@
local Sqlite = require "frecency.database.sqlite"
local Native = require "frecency.database.native"
local Database = require "frecency.database"
local EntryMaker = require "frecency.entry_maker"
local FS = require "frecency.fs"
local Migrator = require "frecency.migrator"
local Picker = require "frecency.picker"
local Recency = require "frecency.recency"
local WebDevicons = require "frecency.web_devicons"
local sqlite_module = require "frecency.sqlite"
local os_util = require "frecency.os_util"
local log = require "plenary.log"
@ -16,7 +13,6 @@ local log = require "plenary.log"
---@field private database FrecencyDatabase
---@field private entry_maker FrecencyEntryMaker
---@field private fs FrecencyFS
---@field private migrator FrecencyMigrator
---@field private picker FrecencyPicker
---@field private recency FrecencyRecency
local Frecency = {}
@ -34,7 +30,6 @@ local Frecency = {}
---@field show_filter_column boolean|string[]|nil default: true
---@field show_scores boolean? default: false
---@field show_unindexed boolean? default: true
---@field use_sqlite boolean? default: false
---@field workspace_scan_cmd "LUA"|string[]|nil default: nil
---@field workspaces table<string, string>? default: {}
@ -56,23 +51,12 @@ Frecency.new = function(opts)
show_filter_column = true,
show_scores = false,
show_unindexed = true,
use_sqlite = false,
workspace_scan_cmd = nil,
workspaces = {},
}, opts or {})
local self = setmetatable({ buf_registered = {}, config = config }, { __index = Frecency })--[[@as Frecency]]
self.fs = FS.new { ignore_patterns = config.ignore_patterns }
local Database
if not self.config.use_sqlite then
Database = Native
elseif not sqlite_module.can_use then
self:warn "use_sqlite = true, but sqlite module can not be found. It fallbacks to native code."
Database = Native
else
self:warn "SQLite mode is deprecated."
Database = Sqlite
end
self.database = Database.new(self.fs, { root = config.db_root })
local web_devicons = WebDevicons.new(not config.disable_devicons)
self.entry_maker = EntryMaker.new(self.fs, web_devicons, {
@ -81,7 +65,6 @@ Frecency.new = function(opts)
})
local max_count = config.max_timestamps > 0 and config.max_timestamps or 10
self.recency = Recency.new { max_count = max_count }
self.migrator = Migrator.new(self.fs, self.recency, self.config.db_root)
return self
end
@ -104,10 +87,6 @@ function Frecency:setup()
self:validate_database()
end
vim.api.nvim_create_user_command("FrecencyMigrateDB", function()
self:migrate_database()
end, { desc = "Migrate DB telescope-frecency to native code" })
vim.api.nvim_create_user_command("FrecencyDelete", function(info)
local path_string = info.args == "" and "%:p" or info.args
local path = vim.fn.expand(path_string) --[[@as string]]
@ -157,17 +136,10 @@ end
---@private
---@return nil
function Frecency:assert_db_entries()
if self.database:has_entry() then
return
elseif not self.config.use_sqlite and sqlite_module.can_use then
local sqlite = Sqlite.new(self.fs, { root = self.config.db_root })
if sqlite:has_entry() then
self:migrate_database(false, true)
return
end
end
if not self.database:has_entry() then
self.database:insert_files(vim.v.oldfiles)
self:notify("Imported %d entries from oldfiles.", #vim.v.oldfiles)
end
end
---@private
@ -213,45 +185,6 @@ function Frecency:register(bufnr, datetime)
self.buf_registered[bufnr] = true
end
---@param to_sqlite boolean?
---@param silently boolean?
---@return nil
function Frecency:migrate_database(to_sqlite, silently)
local function migrate()
if not sqlite_module.can_use then
self:error "sqlite.lua is unavailable"
elseif to_sqlite then
self.migrator:to_sqlite()
self:notify "Migration is finished successfully."
else
self.migrator:to_v1()
self:notify "Migration is finished successfully. You can remove sqlite.lua from dependencies."
end
end
if silently then
migrate()
return
end
local prompt = to_sqlite and "Migrate the DB into SQLite from native code?"
or "Migrate the DB into native code from SQLite?"
vim.ui.select({ "y", "n" }, {
prompt = prompt,
---@param item "y"|"n"
---@return string
format_item = function(item)
return item == "y" and "Yes, Migrate it." or "No. Do nothing."
end,
}, function(item)
if item == "y" then
migrate()
else
self:notify "Migration aborted"
end
end)
end
---@param path string
---@return nil
function Frecency:delete(path)

83
lua/frecency/migrator.lua
View File

@ -1,83 +0,0 @@
local Sqlite = require "frecency.database.sqlite"
local Native = require "frecency.database.native"
local wait = require "frecency.wait"
---@class FrecencyMigrator
---@field fs FrecencyFS
---@field recency FrecencyRecency
---@field root string
local Migrator = {}
---@param fs FrecencyFS
---@param recency FrecencyRecency
---@param root string
---@return FrecencyMigrator
Migrator.new = function(fs, recency, root)
return setmetatable({ fs = fs, recency = recency, root = root }, { __index = Migrator })
end
---@return nil
function Migrator:to_v1()
local native = Native.new(self.fs, { root = self.root })
native.table = self:v1_table_from_sqlite()
wait(function()
native:save()
end)
end
---@return nil
function Migrator:to_sqlite()
local sqlite = Sqlite.new(self.fs, { root = self.root })
local native = Native.new(self.fs, { root = self.root })
for path, record in pairs(native.table.records) do
local file_id = sqlite.sqlite.files:insert { path = path, count = record.count }
sqlite.sqlite.timestamps:insert(vim.tbl_map(function(timestamp)
return { file_id = file_id, timestamp = ('julianday(datetime(%d, "unixepoch"))'):format(timestamp) }
end, record.timestamps))
end
end
---@private
---@return FrecencyDatabaseNativeTable
function Migrator:v1_table_from_sqlite()
local sqlite = Sqlite.new(self.fs, { root = self.root })
---@type FrecencyDatabaseNativeTable
local tbl = { version = "v1", records = {} }
local files = sqlite.sqlite.files:get {} --[[@as FrecencyFile[] ]]
---@type table<integer,string>
local path_map = {}
for _, file in ipairs(files) do
tbl.records[file.path] = { count = file.count, timestamps = { 0 } }
path_map[file.id] = file.path
end
-- local timestamps = sqlite.sqlite.timestamps:get { keys = { "id", "file_id", epoch = "unixepoch(timestamp)" } } --[[@as FrecencyTimestamp[] ]]
local timestamps = sqlite.sqlite.timestamps:get {
keys = { "id", "file_id", epoch = "cast(strftime('%s', timestamp) as integer)" },
} --[[@as FrecencyTimestamp[] ]]
table.sort(timestamps, function(a, b)
return a.id < b.id
end)
for _, timestamp in ipairs(timestamps) do
local path = path_map[timestamp.file_id]
if path then
local record = tbl.records[path]
if record then
if #record.timestamps == 1 and record.timestamps[1] == 0 then
record.timestamps = {}
end
---@diagnostic disable-next-line: undefined-field
table.insert(record.timestamps, timestamp.epoch)
if #record.timestamps > self.recency.config.max_count then
local new_table = {}
for i = #record.timestamps - self.recency.config.max_count + 1, #record.timestamps do
table.insert(new_table, record.timestamps[i])
end
record.timestamps = new_table
end
end
end
end
return tbl
end
return Migrator

17
lua/frecency/sqlite.lua
View File

@ -1,17 +0,0 @@
---@class FrecencySqlite
---@field can_use boolean
---@field lib sqlite_lib
---@overload fun(opts: table): FrecencySqliteDB
return setmetatable({}, {
__index = function(_, k)
if k == "lib" then
return require("sqlite").lib
elseif k == "can_use" then
return not not vim.F.npcall(require, "sqlite")
end
end,
__call = function(_, opts)
return require "sqlite"(opts)
end,
}) --[[@as FrecencySqlite]]

20
lua/frecency/tests/native_spec.lua → lua/frecency/tests/database_spec.lua
View File

@ -1,41 +1,41 @@
local FS = require "frecency.fs"
local Native = require "frecency.database.native"
local Database = require "frecency.database"
local async = require "plenary.async" --[[@as PlenaryAsync]]
local util = require "frecency.tests.util"
async.tests.add_to_env()
local function with_native(f)
local function with_database(f)
local fs = FS.new { ignore_patterns = {} }
local dir, close = util.tmpdir()
dir:joinpath("file_frecency.bin"):touch()
return function()
local native = Native.new(fs, { root = dir.filename })
f(native)
local database = Database.new(fs, { root = dir.filename })
f(database)
close()
end
end
local function save_and_load(native, tbl, datetime)
native:raw_save(util.v1_table(tbl))
local function save_and_load(database, tbl, datetime)
database:raw_save(util.v1_table(tbl))
async.util.sleep(100)
local entries = native:get_entries(nil, datetime)
local entries = database:get_entries(nil, datetime)
table.sort(entries, function(a, b)
return a.path < b.path
end)
return entries
end
a.describe("frecency.database.native", function()
a.describe("frecency.database", function()
a.describe("updated by another process", function()
a.it(
"returns valid entries",
with_native(function(native)
with_database(function(database)
assert.are.same(
{
{ path = "hoge1.txt", count = 1, ages = { 60 } },
{ path = "hoge2.txt", count = 1, ages = { 60 } },
},
save_and_load(native, {
save_and_load(database, {
["hoge1.txt"] = { count = 1, timestamps = { "2023-08-21T00:00:00+0000" } },
["hoge2.txt"] = { count = 1, timestamps = { "2023-08-21T00:00:00+0000" } },
}, "2023-08-21T01:00:00+0000")

26
lua/frecency/tests/frecency_spec.lua
View File

@ -1,19 +1,21 @@
---@diagnostic disable: invisible
-- HACK: This is needed because plenary.test_harness resets &rtp.
-- https://github.com/nvim-lua/plenary.nvim/blob/663246936325062427597964d81d30eaa42ab1e4/lua/plenary/test_harness.lua#L86-L86
vim.opt.runtimepath:append(vim.env.TELESCOPE_PATH)
---@diagnostic disable: invisible, undefined-field
local Frecency = require "frecency.frecency"
local Picker = require "frecency.picker"
local util = require "frecency.tests.util"
local log = require "plenary.log"
local Path = require "plenary.path"
local use_sqlite
---@param files string[]
---@param callback fun(frecency: Frecency, finder: FrecencyFinder, dir: PlenaryPath): nil
---@return nil
local function with_files(files, callback)
local dir, close = util.make_tree(files)
log.debug { db_root = dir.filename }
local frecency = Frecency.new { db_root = dir.filename, use_sqlite = use_sqlite }
local frecency = Frecency.new { db_root = dir.filename }
frecency.picker = Picker.new(
frecency.database,
frecency.entry_maker,
@ -89,8 +91,6 @@ local function with_fake_vim_ui_select(choice, callback)
end
describe("frecency", function()
local function test(db)
describe(db, function()
describe("register", function()
describe("when opening files", function()
with_files({ "hoge1.txt", "hoge2.txt" }, function(frecency, finder, dir)
@ -232,9 +232,7 @@ describe("frecency", function()
describe("when with not force", function()
describe("when files are unlinked but it is less than threshold", function()
with_files(
{ "hoge1.txt", "hoge2.txt", "hoge3.txt", "hoge4.txt", "hoge5.txt" },
function(frecency, finder, dir)
with_files({ "hoge1.txt", "hoge2.txt", "hoge3.txt", "hoge4.txt", "hoge5.txt" }, function(frecency, finder, dir)
local register = make_register(frecency, dir)
register("hoge1.txt", "2023-07-29T00:00:00+09:00")
register("hoge2.txt", "2023-07-29T00:01:00+09:00")
@ -259,8 +257,7 @@ describe("frecency", function()
{ count = 1, path = filepath(dir, "hoge5.txt"), score = 10 },
}, results)
end)
end
)
end)
end)
describe("when files are unlinked and it is more than threshold", function()
@ -423,11 +420,4 @@ describe("frecency", function()
end)
end)
end)
end)
end
use_sqlite = true
test "sqlite"
use_sqlite = false
test "native"
end)

145
lua/frecency/tests/migrator_spec.lua
View File

@ -1,145 +0,0 @@
---@diagnostic disable: undefined-field
local Migrator = require "frecency.migrator"
local FS = require "frecency.fs"
local Recency = require "frecency.recency"
local Sqlite = require "frecency.database.sqlite"
local Native = require "frecency.database.native"
local util = require "frecency.tests.util"
local wait = require "frecency.wait"
-- TODO: replace this with vim.system
local Job = require "plenary.job"
---@param callback fun(migrator: FrecencyMigrator, sqlite: FrecencyDatabase): nil
---@return nil
local function with(callback)
local dir, close = util.tmpdir()
local recency = Recency.new { max_count = 2 }
local fs = FS.new { ignore_patterns = {} }
local migrator = Migrator.new(fs, recency, dir.filename)
local sqlite = Sqlite.new(fs, { root = dir.filename })
callback(migrator, sqlite)
close()
end
local function strptime(iso8601)
local result = vim.fn.strptime("%FT%T%z", iso8601)
return result ~= 0 and result or nil
end
-- NOTE: Windows has no strptime
local function time_piece(iso8601)
local stdout, code =
Job:new({ "perl", "-MTime::Piece", "-e", "print Time::Piece->strptime('" .. iso8601 .. "', '%FT%T%z')->epoch" })
:sync(30000)
return code == 0 and tonumber(stdout[1]) or nil
end
---@param source table<string,{ count: integer, timestamps: string[] }>
local function v1_table(source)
local records = {}
for path, record in pairs(source) do
local timestamps = {}
for _, timestamp in ipairs(record.timestamps) do
local iso8601 = timestamp .. "+0000"
table.insert(timestamps, strptime(iso8601) or time_piece(iso8601))
end
records[path] = { count = record.count, timestamps = timestamps }
end
return { version = "v1", records = records }
end
describe("migrator", function()
describe("to_v1", function()
describe("when with simple database", function()
with(function(migrator, sqlite)
for _, path in ipairs { "hoge1.txt", "hoge2.txt" } do
sqlite:update(path, migrator.recency.config.max_count, "2023-08-21T00:00:00")
end
migrator:to_v1()
local native = Native.new(migrator.fs, { root = migrator.root })
it("has converted into a valid table", function()
assert.are.same(
native.table,
v1_table {
["hoge1.txt"] = { count = 1, timestamps = { "2023-08-21T00:00:00" } },
["hoge2.txt"] = { count = 1, timestamps = { "2023-08-21T00:00:00" } },
}
)
end)
end)
end)
describe("when with more large database", function()
with(function(migrator, sqlite)
for i, path in ipairs {
"hoge1.txt",
"hoge1.txt",
"hoge1.txt",
"hoge1.txt",
"hoge2.txt",
"hoge2.txt",
"hoge2.txt",
"hoge3.txt",
"hoge3.txt",
"hoge4.txt",
} do
sqlite:update(path, migrator.recency.config.max_count, ("2023-08-21T00:%02d:00"):format(i))
end
migrator:to_v1()
local native = Native.new(migrator.fs, { root = migrator.root })
it("has converted into a valid table", function()
assert.are.same(
native.table,
v1_table {
["hoge1.txt"] = { count = 4, timestamps = { "2023-08-21T00:03:00", "2023-08-21T00:04:00" } },
["hoge2.txt"] = { count = 3, timestamps = { "2023-08-21T00:06:00", "2023-08-21T00:07:00" } },
["hoge3.txt"] = { count = 2, timestamps = { "2023-08-21T00:08:00", "2023-08-21T00:09:00" } },
["hoge4.txt"] = { count = 1, timestamps = { "2023-08-21T00:10:00" } },
}
)
end)
end)
end)
end)
describe("to_sqlite", function()
with(function(migrator, sqlite)
local native = Native.new(migrator.fs, { root = migrator.root })
native.table = v1_table {
["hoge1.txt"] = { count = 4, timestamps = { "2023-08-21T00:03:00", "2023-08-21T00:04:00" } },
["hoge2.txt"] = { count = 3, timestamps = { "2023-08-21T00:06:00", "2023-08-21T00:07:00" } },
["hoge3.txt"] = { count = 2, timestamps = { "2023-08-21T00:08:00", "2023-08-21T00:09:00" } },
["hoge4.txt"] = { count = 1, timestamps = { "2023-08-21T00:10:00" } },
}
wait(function()
native:save()
end)
migrator:to_sqlite()
sqlite.sqlite.db:open()
local records = sqlite.sqlite.db:eval [[
select
f.path,
f.count,
datetime(strftime('%s', t.timestamp), 'unixepoch') datetime
from timestamps t
join files f
on f.id = t.file_id
order by path, datetime
]]
it("has converted into a valid DB", function()
assert.are.same(records, {
{ path = "hoge1.txt", count = 4, datetime = "2023-08-21 00:03:00" },
{ path = "hoge1.txt", count = 4, datetime = "2023-08-21 00:04:00" },
{ path = "hoge2.txt", count = 3, datetime = "2023-08-21 00:06:00" },
{ path = "hoge2.txt", count = 3, datetime = "2023-08-21 00:07:00" },
{ path = "hoge3.txt", count = 2, datetime = "2023-08-21 00:08:00" },
{ path = "hoge3.txt", count = 2, datetime = "2023-08-21 00:09:00" },
{ path = "hoge4.txt", count = 1, datetime = "2023-08-21 00:10:00" },
})
end)
end)
end)
end)

4
lua/frecency/tests/minimal.lua
View File

@ -4,10 +4,6 @@ end
if not vim.env.TELESCOPE_PATH then
error "set $TELESCOPE_PATH to find telescope.nvim"
end
if not vim.env.SQLITE_PATH then
error "set $SQLITE_PATH to find telescope.nvim"
end
vim.opt.runtimepath:append(vim.env.PLENARY_PATH)
vim.opt.runtimepath:append(vim.env.TELESCOPE_PATH)
vim.opt.runtimepath:append(vim.env.SQLITE_PATH)
vim.cmd.runtime "plugin/plenary.vim"

43
lua/frecency/types.lua
View File

@ -1,48 +1,5 @@
---@diagnostic disable: unused-local, missing-return
-- NOTE: types below are borrowed from sqlite.lua
---@class sqlite_db @Main sqlite.lua object.
---@field uri string: database uri. it can be an environment variable or an absolute path. default ":memory:"
---@field opts sqlite_opts: see https://www.sqlite.org/pragma.html |sqlite_opts|
---@field conn sqlite_blob: sqlite connection c object.
---@field db sqlite_db: reference to fallback to when overwriting |sqlite_db| methods (extended only).
---@class sqlite_query_update @Query fileds used when calling |sqlite:update| or |sqlite_tbl:update|
---@field where table: filter down values using key values.
---@field set table: key and value to updated.
---@class sqlite_query_select @Query fileds used when calling |sqlite:select| or |sqlite_tbl:get|
---@field where table? filter down values using key values.
---@field keys table? keys to include. (default all)
---@field join table? (TODO: support)
---@field order_by table? { asc = "key", dsc = {"key", "another_key"} }
---@field limit number? the number of result to limit by
---@field contains table? for sqlite glob ex. { title = "fix*" }
---@alias sqlite_query_delete table<string, any>
---@generic T
---@alias sqlite_map_func fun(self: sqlite_tbl, mapper: fun(entry: table): T?): T[]
---@class sqlite_tbl @Main sql table class
---@field db sqlite_db: sqlite.lua database object.
---@field name string: table name.
---@field mtime number: db last modified time.
---@field count fun(self: sqlite_tbl): integer
---@field insert fun(self: sqlite_tbl, rows: table<string, any>|table<string, any>[]): integer
---@field update fun(self: sqlite_tbl, specs: sqlite_query_update): boolean
---@field get fun(self: sqlite_tbl, query: sqlite_query_select): table[]
---@field remove fun(self: sqlite_tbl, where: sqlite_query_delete): boolean
---@field map sqlite_map_func
---@class sqlite_opts @Sqlite3 Options (TODO: add sqlite option fields and description)
---@class sqlite_blob @sqlite3 blob object
---@class sqlite_lib
---@field cast fun(source: integer, as: string): string
---@field julianday fun(timestring: string?): integer
-- NOTE: types are borrowed from plenary.nvim
---@class PlenaryPath

0
lua/frecency/database/native/watcher.lua → lua/frecency/watcher.lua
View File

6
lua/telescope/_extensions/frecency.lua
View File

@ -1,14 +1,8 @@
local frecency = require "frecency"
local sqlite = require "frecency.sqlite"
return require("telescope").register_extension {
setup = frecency.setup,
health = function()
if sqlite.can_use then
vim.health.ok "sqlite.lua installed."
else
vim.health.info "sqlite.lua is required when use_sqlite = true"
end
if vim.F.npcall(require, "nvim-web-devicons") then
vim.health.ok "nvim-web-devicons installed."
else