feat!: make use_sqlite = false be the default (#148)

* refactor: load lazily sqlite to get filename

* feat: add logic to start migration automatically

* feat: make use_sqlite = false be the default

* docs: update README for removing dependencies

* feat: migrate silently when use_sqlite = false

* chore: remove `SQLite3` from messages

* test: run again max to 5 times in Windows

.github/workflows/ci.yml

@@ -70,7 +70,12 @@ jobs:
           TEST_DIR=lua/frecency/tests/
           MINIMAL_LUA=${TEST_DIR}minimal.lua
           NVIM=$(perl -e '$_ = $ENV{EXE}; s,\\,/,g; print')
-          $NVIM --headless --clean -u $MINIMAL_LUA -c "PlenaryBustedDirectory $TEST_DIR {minimal_init = '$MINIMAL_LUA', timeout = 180000, sequential = true}"
+          attempts=0
+          while ! $NVIM --headless --clean -u $MINIMAL_LUA -c "PlenaryBustedDirectory $TEST_DIR {minimal_init = '$MINIMAL_LUA', timeout = 180000, sequential = true}"; do
+            if (( attempts++ > 5 )); then
+              exit 1
+            fi
+          done
         if: matrix.os == 'windows-latest'
       - name: Type Check Code Base
         uses: mrcjkb/lua-typecheck-action@v0.2.0

README.md

@@ -1,10 +1,19 @@
 # telescope-frecency.nvim
 
-A [telescope.nvim](https://github.com/nvim-telescope/telescope.nvim) extension that offers intelligent prioritization when selecting files from your editing history.
+A [telescope.nvim][] extension that offers intelligent prioritization when
+selecting files from your editing history.
 
-Using an implementation of Mozilla's [Frecency algorithm](https://web.archive.org/web/20210421120120/https://developer.mozilla.org/en-US/docs/Mozilla/Tech/Places/Frecency_algorithm) (used in [Firefox's address bar](https://support.mozilla.org/en-US/kb/address-bar-autocomplete-firefox)), files edited _frecently_ are given higher precedence in the list index.
+[telescope.nvim]: https://github.com/nvim-telescope/telescope.nvim
 
-As the extension learns your editing habits over time, the sorting of the list is dynamically altered to prioritize the files you're likely to need.
+Using an implementation of Mozilla's [Frecency algorithm][] (used in [Firefox's
+address bar][]), files edited _frecently_ are given higher precedence in the
+list index.
+
+[Frecency algorithm]: https://web.archive.org/web/20210421120120/https://developer.mozilla.org/en-US/docs/Mozilla/Tech/Places/Frecency_algorithm
+[Firefox's address bar]: https://support.mozilla.org/en-US/kb/address-bar-autocomplete-firefox
+
+As the extension learns your editing habits over time, the sorting of the list
+is dynamically altered to prioritize the files you're likely to need.
 
 <img src="https://raw.githubusercontent.com/sunjon/images/master/gh_readme_telescope_frecency.png" alt="screenshot" width="800"/>
 
@@ -12,11 +21,14 @@ As the extension learns your editing habits over time, the sorting of the list i
 
 ## Frecency: Sorting by 'frequency' _and_ 'recency'
 
-'Frecency' is a score given to each unique file indexed in a file history database.
+'Frecency' is a score given to each unique file indexed in a file history
+database.
 
-A timestamp is recorded once per session when a file is first loaded into a buffer.
+A timestamp is recorded once per session when a file is first loaded into a
+buffer.
 
-The score is calculated using the age of the 10 (customizable) most recent timestamps and the total amount of times that the file has been loaded:
+The score is calculated using the age of the 10 (customizable) most recent
+timestamps and the total amount of times that the file has been loaded:
 
 ### Recency values (per timestamp)
 
@@ -36,36 +48,46 @@ score = frequency * recency_score / max_number_of_timestamps
 ```
 ## What about files that are neither 'frequent' _or_ 'recent' ?
 
-Frecency naturally works best for indexed files that have been given a reasonably high score.
+Frecency naturally works best for indexed files that have been given a
+reasonably high score.
 
-New projects or rarely used files with generic names either don't get listed at all or can be buried under results with a higher score.
+New projects or rarely used files with generic names either don't get listed at
+all or can be buried under results with a higher score.
 
 Frecency tackles this with *Workspace Filters*:
 
 <img src="https://raw.githubusercontent.com/sunjon/images/master/frecency_workspace_folders.gif" alt="screenshot" width="800"/>
 
-The workspace filter feature enables you to select from user defined _filter tags_ that map to a directory or collection of directories.
+The workspace filter feature enables you to select from user defined _filter
-Filters are applied by entering `:workspace_tag:` anywhere in the query.
+tags_ that map to a directory or collection of directories. Filters are applied
-Filter name completion is available by pressing `<Tab>` after the first `:` character.
+by entering `:workspace_tag:` anywhere in the query. Filter name completion is
+available by pressing `<Tab>` after the first `:` character.
 
-When a filter is applied, results are reduced to entries whose path is a descendant of the workspace directory.
+When a filter is applied, results are reduced to entries whose path is a
-The indexed results are optionally augmented with a listing of _all_ files found in a recurssive search of the workspace directory.
+descendant of the workspace directory. The indexed results are optionally
-Non-indexed files are given a score of zero and appear below the 'frecent' entries.
+augmented with a listing of _all_ files found in a recurssive search of the
-When a non-indexed file is opened, it gains a score value and is available in future 'frecent' search results.
+workspace directory. Non-indexed files are given a score of zero and appear
+below the 'frecent' entries. When a non-indexed file is opened, it gains a
+score value and is available in future 'frecent' search results.
 
-If the active buffer (prior to the finder being launched) is attached to an LSP server, an automatic `LSP` tag is available, which maps to the workspace directories provided by the language server.
+If the active buffer (prior to the finder being launched) is attached to an LSP
+server, an automatic `LSP` tag is available, which maps to the workspace
+directories provided by the language server.
 
 
 ## Requirements
 
 - [telescope.nvim](https://github.com/nvim-telescope/telescope.nvim) (required)
-- [sqlite.lua][] (required)
 - [nvim-web-devicons](https://github.com/kyazdani42/nvim-web-devicons) (optional)
 
-[sqlite.lua]: https://github.com/kkharji/sqlite.lua
+**NOTE:** The former version of this plugin has used [SQLite3][] database to
+store timestamps and file records. But the current build uses Lua native code
+to store them, so you can now remove [sqlite.lua][] from dependencies. See
+[*Remove dependency for sqlite.lua*][remove-sqlite] for the detail.
 
-Timestamps and file records are stored in an [SQLite3](https://www.sqlite.org/index.html) database for persistence and speed.
+[SQLite3]: https://www.sqlite.org/index.html
-This plugin uses `sqlite.lua` to perform the database transactions.
+[sqlite.lua]: https://github.com/kkharji/sqlite.lua
+[remove-sqlite]: #user-content-remove-dependency-for-sqlitelua
 
 ## Installation
 
@@ -77,7 +99,6 @@ use {
   config = function()
     require("telescope").load_extension "frecency"
   end,
-  requires = { "kkharji/sqlite.lua" },
 }
 ```
 
@@ -89,11 +110,12 @@ use {
   config = function()
     require("telescope").load_extension "frecency"
   end,
-  dependencies = { "kkharji/sqlite.lua" },
 }
 ```
 
-If no database is found when running Neovim with the plugin installed, a new one is created and entries from `shada` `v:oldfiles` are automatically imported.
+If no database is found when running Neovim with the plugin installed, a new
+one is created and entries from `shada` `v:oldfiles` are automatically
+imported.
 
 ## Usage
 
@@ -104,7 +126,7 @@ If no database is found when running Neovim with the plugin installed, a new one
 or to map to a key:
 
 ```lua
-vim.api.nvim_set_keymap("n", "<leader><leader>", "<Cmd>Telescope frecency<CR>")
+vim.keymap.set("n", "<leader><leader>", "<Cmd>Telescope frecency<CR>")
 ```
 
 Use a specific workspace tag:
@@ -116,11 +138,12 @@ Use a specific workspace tag:
 or
 
 ```lua
-vim.api.nvim_set_keymap("n", "<leader><leader>", "<Cmd>Telescope frecency workspace=CWD<CR>")
+vim.keymap.set("n", "<leader><leader>", "<Cmd>Telescope frecency workspace=CWD<CR>")
 ```
 
-Filter tags are applied by typing the `:tag:` name (adding surrounding colons) in the finder query.
+Filter tags are applied by typing the `:tag:` name (adding surrounding colons)
-Entering `:<Tab>` will trigger omnicompletion for available tags.
+in the finder query. Entering `:<Tab>` will trigger omnicompletion for
+available tags.
 
 ## Configuration
 
@@ -128,12 +151,14 @@ See [default configuration](https://github.com/nvim-telescope/telescope.nvim#tel
 
 - `db_root` (default: `vim.fn.stdpath "data"`)
 
-  Path to parent directory of custom database location.
+  Path to parent directory of custom database location. Defaults to
-  Defaults to `$XDG_DATA_HOME/nvim` if unset.
+  `$XDG_DATA_HOME/nvim` if unset.
 
 - `default_workspace` (default: `nil`)
 
-  Default workspace tag to filter by e.g. `'CWD'` to filter by default to the current directory. Can be overridden at query time by specifying another filter like `':*:'`.
+  Default workspace tag to filter by e.g. `'CWD'` to filter by default to the
+  current directory. Can be overridden at query time by specifying another
+  filter like `':*:'`.
 
 - `disable_devicons` (default: `false`)
 
@@ -141,17 +166,23 @@ See [default configuration](https://github.com/nvim-telescope/telescope.nvim#tel
 
 - `ignore_patterns` (default: `{ "*.git/*", "*/tmp/*", "term://*" }`)
 
-  Patterns in this table control which files are indexed (and subsequently which you'll see in the finder results).
+  Patterns in this table control which files are indexed (and subsequently
+  which you'll see in the finder results).
 
 - `max_timestamps` (default: `10`)
 
-  Set the max count of timestamps DB keeps when you open files. It ignores the value and use `10` if you set less than or equal to `0`.
+  Set the max count of timestamps DB keeps when you open files. It ignores the
+  value and use `10` if you set less than or equal to `0`.
 
-  **CAUTION** When you reduce the value of this option, it removes old timestamps when you open the file. It is reasonable to set this value more than or equal to the default value: `10`.
+  **CAUTION** When you reduce the value of this option, it removes old
+  timestamps when you open the file. It is reasonable to set this value more
+  than or equal to the default value: `10`.
 
 - `show_filter_column` (default: `true`)
 
-  Show the path of the active filter before file paths. In default, it uses the tail of paths for `'LSP'` and `'CWD'` tags. You can configure this by setting a table for this option.
+  Show the path of the active filter before file paths. In default, it uses the
+  tail of paths for `'LSP'` and `'CWD'` tags. You can configure this by setting
+  a table for this option.
 
    ```lua
    -- show the tail for "LSP", "CWD" and "FOO"
@@ -160,21 +191,24 @@ See [default configuration](https://github.com/nvim-telescope/telescope.nvim#tel
 
 - `show_scores` (default : `false`)
 
-  To see the scores generated by the algorithm in the results, set this to `true`.
+  To see the scores generated by the algorithm in the results, set this to
+  `true`.
 
 - `show_unindexed` (default: `true`)
 
-    Determines if non-indexed files are included in workspace filter results.
+  Determines if non-indexed files are included in workspace filter results.
 
-- `use_sqlite` (default: `true`) ***experimental feature***
+- `use_sqlite` (default: `false`)
 
-  Use [sqlite.lua] `true` or native code `false`. See [*Remove dependency for sqlite.lua*](#user-content-remove-dependency-for-sqlitelua) for the detail.
+  Use [sqlite.lua][] with `true` or native code with `false`. See [*Remove
+  dependency for sqlite.lua*][remove-sqlite] for the detail.
 
 - `workspaces` (default: `{}`)
 
-    This table contains mappings of `workspace_tag` -> `workspace_directory`
+  This table contains mappings of `workspace_tag` -> `workspace_directory`. The
-    The key corresponds to the `:tag_name` used to select the filter in queries.
+  key corresponds to the `:tag_name` used to select the filter in queries. The
-    The value corresponds to the top level directory by which results will be filtered.
+  value corresponds to the top level directory by which results will be
+  filtered.
 
 ### Example Configuration:
 
@@ -182,10 +216,10 @@ See [default configuration](https://github.com/nvim-telescope/telescope.nvim#tel
 telescope.setup {
   extensions = {
     frecency = {
-      db_root = "home/my_username/path/to/db_root",
+      db_root = "/home/my_username/path/to/db_root",
       show_scores = false,
       show_unindexed = true,
-      ignore_patterns = {"*.git/*", "*/tmp/*"},
+      ignore_patterns = { "*.git/*", "*/tmp/*" },
       disable_devicons = false,
       workspaces = {
         ["conf"]    = "/home/my_username/.config",
@@ -202,19 +236,25 @@ telescope.setup {
 
 ### Location
 
-The default location for the database is `$XDG_DATA_HOME/nvim` (eg `~/.local/share/nvim/` on linux).
+The default location for the database is `$XDG_DATA_HOME/nvim` (eg
-This can be configured with the `db_root` config option.
+`~/.local/share/nvim/` on linux). This can be configured with the `db_root`
+config option.
 
 ### Maintainance
 
 By default, frecency will prune files that no longer exist from the database.
-In certain workflows, switching branches in a repository, that behaviour might not be desired.
+In certain workflows, switching branches in a repository, that behaviour might
-The following configuration control this behaviour:
+not be desired. The following configuration control this behaviour:
 
-`db_safe_mode` - When this is enabled, the user will be prompted before any entries are removed from the database.
+<dl>
-`auto_validate` - When this to false, stale entries will never be automatically removed.
+<dt><code>db_safe_mode</code></dt>
+<dd>When this is enabled, the user will be prompted before any entries are removed from the database.</dd>
+<dt><code>auto_validate</code></dt>
+<dd>When this to false, stale entries will never be automatically removed.</dd>
+</dl>
 
-The command `FrecencyValidate` can be used to clean the database when `auto_validate` is disabled.
+The command `FrecencyValidate` can be used to clean the database when
+`auto_validate` is disabled.
 
 ```vim
 " clean DB
@@ -225,13 +265,19 @@ The command `FrecencyValidate` can be used to clean the database when `auto_vali
 
 ### Remove dependency for [sqlite.lua][]
 
-***This is an experimental feature.***
+The former version of this plugin has used SQLite3 library to store data. When
+you upgrade from such version, Neovim will silently migrate DB and inform that
+you can remove `sqlite.lua` from dependencies.
 
-In default, it uses SQLite3 library to access the DB. When `use_sqlite` option is set to `false`, it stores the whole data and saves them with encoding by `string.dump()` Lua function.
+| made by default | made by `sqlite.lua` |
+|--|--|
+| `~/.local/share/nvim/file_frecency.bin` | `~/.local/share/nvim/file_frecency.sqlite3` |
 
-With this, we can remove the dependency for [sqlite.lua][] and obtain faster speed to open `:Telescope frecency`.
+The DB file will be migrated into a filename above, and old file (SQLite3
+version) will still remain. If you still want to use SQLite3 version, set
+`use_sqlite = true`.
 
-You can migrate from SQLite DB into native code by `:FrecencyMigrateDB` command. It converts data into native code, but does not delete the existent SQLite DB. You can use old SQLite logic by `use_sqlite = true` again.
+Also you can explicitly migrate DB by calling `:FrecencyMigrateDB` command.
 
 ## Highlight Groups
 

lua/frecency/database.lua

@@ -8,6 +8,7 @@
 
 ---@class FrecencyDatabase
 ---@field config FrecencyDatabaseConfig
+---@field filename string
 ---@field has_entry fun(): boolean
 ---@field new fun(fs: FrecencyFS, config: FrecencyDatabaseConfig): FrecencyDatabase
 ---@field protected fs FrecencyFS

lua/frecency/database/native.lua

@@ -7,7 +7,6 @@ local Path = require "plenary.path" --[[@as PlenaryPath]]
 
 ---@class FrecencyDatabaseNative: FrecencyDatabase
 ---@field version "v1"
----@field filename string
 ---@field file_lock FrecencyFileLock
 ---@field table FrecencyDatabaseNativeTable
 local Native = {}

lua/frecency/database/sqlite.lua

@@ -1,5 +1,6 @@
 local sqlite = require "frecency.sqlite"
 local log = require "plenary.log"
+local Path = require "plenary.path" --[[@as PlenaryPath]]
 
 ---@class FrecencySqliteDB: sqlite_db
 ---@field files sqlite_tbl
@@ -25,20 +26,32 @@ local Sqlite = {}
 ---@param config FrecencyDatabaseConfig
 ---@return FrecencyDatabaseSqlite
 Sqlite.new = function(fs, config)
-  local lib = sqlite.lib
   local self = setmetatable(
     { config = config, buf_registered_flag_name = "telescope_frecency_registered", fs = fs },
     { __index = Sqlite }
   )
-  self.sqlite = sqlite {
+  self.filename = Path.new(self.config.root, "file_frecency.sqlite3").filename
-    uri = self.config.root .. "/file_frecency.sqlite3",
+  self.sqlite = setmetatable({}, {
-    files = { id = true, count = { "integer", default = 1, required = true }, path = "string" },
+    __index = function(this, key)
-    timestamps = {
+      if not rawget(this, "instance") then
-      id = true,
+        local lib = sqlite.lib
-      file_id = { "integer", reference = "files.id", on_delete = "cascade" },
+        rawset(
-      timestamp = { "real", default = lib.julianday "now" },
+          this,
-    },
+          "instance",
-  }
+          sqlite {
+            uri = self.filename,
+            files = { id = true, count = { "integer", default = 1, required = true }, path = "string" },
+            timestamps = {
+              id = true,
+              file_id = { "integer", reference = "files.id", on_delete = "cascade" },
+              timestamp = { "real", default = lib.julianday "now" },
+            },
+          }
+        )
+      end
+      return rawget(this, "instance")[key]
+    end,
+  })
   return self
 end
 

lua/frecency/frecency.lua

@@ -33,7 +33,7 @@ local Frecency = {}
 ---@field show_filter_column boolean|string[]|nil default: true
 ---@field show_scores boolean? default: false
 ---@field show_unindexed boolean? default: true
----@field use_sqlite boolean? default: true
+---@field use_sqlite boolean? default: false
 ---@field workspaces table<string, string>? default: {}
 
 ---@param opts FrecencyConfig?
@@ -53,7 +53,7 @@ Frecency.new = function(opts)
     show_filter_column = true,
     show_scores = false,
     show_unindexed = true,
-    use_sqlite = true,
+    use_sqlite = false,
     workspaces = {},
   }, opts or {})
   local self = setmetatable({ buf_registered = {}, config = config }, { __index = Frecency })--[[@as Frecency]]
@@ -66,6 +66,7 @@ Frecency.new = function(opts)
     self:warn "use_sqlite = true, but sqlite module can not be found. It fallbacks to native code."
     Database = Native
   else
+    self:warn "SQLite mode is deprecated."
     Database = Sqlite
   end
   self.database = Database.new(self.fs, { root = config.db_root })
@@ -88,10 +89,7 @@ function Frecency:setup()
   vim.api.nvim_set_hl(0, "TelescopeQueryFilter", { link = "WildMenu", default = true })
 
   -- TODO: Should we schedule this after loading shada?
-  if not self.database:has_entry() then
+  self:assert_db_entries()
-    self.database:insert_files(vim.v.oldfiles)
-    self:notify("Imported %d entries from oldfiles.", #vim.v.oldfiles)
-  end
 
   ---@param cmd_info { bang: boolean }
   vim.api.nvim_create_user_command("FrecencyValidate", function(cmd_info)
@@ -142,6 +140,22 @@ function Frecency:complete(findstart, base)
   return self.picker:complete(findstart, base)
 end
 
+---@private
+---@return nil
+function Frecency:assert_db_entries()
+  if self.database:has_entry() then
+    return
+  elseif not self.config.use_sqlite and sqlite_module.can_use then
+    local sqlite = Sqlite.new(self.fs, { root = self.config.db_root })
+    if sqlite:has_entry() then
+      self:migrate_database(false, true)
+      return
+    end
+  end
+  self.database:insert_files(vim.v.oldfiles)
+  self:notify("Imported %d entries from oldfiles.", #vim.v.oldfiles)
+end
+
 ---@private
 ---@param force boolean?
 ---@return nil
@@ -159,7 +173,7 @@ function Frecency:validate_database(force)
     return
   end
   vim.ui.select({ "y", "n" }, {
-    prompt = self:message("remove %d entries from SQLite3 database?", #unlinked),
+    prompt = self:message("remove %d entries from database?", #unlinked),
     ---@param item "y"|"n"
     ---@return string
     format_item = function(item)
@@ -186,10 +200,28 @@ function Frecency:register(bufnr, datetime)
 end
 
 ---@param to_sqlite boolean?
+---@param silently boolean?
 ---@return nil
-function Frecency:migrate_database(to_sqlite)
+function Frecency:migrate_database(to_sqlite, silently)
-  local prompt = to_sqlite and "migrate the DB into SQLite from native code?"
+  local function migrate()
-    or "migrate the DB into native code from SQLite?"
+    if not sqlite_module.can_use then
+      self:error "sqlite.lua is unavailable"
+    elseif to_sqlite then
+      self.migrator:to_sqlite()
+      self:notify "Migration is finished successfully."
+    else
+      self.migrator:to_v1()
+      self:notify "Migration is finished successfully. You can remove sqlite.lua from dependencies."
+    end
+  end
+
+  if silently then
+    migrate()
+    return
+  end
+
+  local prompt = to_sqlite and "Migrate the DB into SQLite from native code?"
+    or "Migrate the DB into native code from SQLite?"
   vim.ui.select({ "y", "n" }, {
     prompt = prompt,
     ---@param item "y"|"n"
@@ -198,20 +230,11 @@ function Frecency:migrate_database(to_sqlite)
       return item == "y" and "Yes, Migrate it." or "No. Do nothing."
     end,
   }, function(item)
-    if item == "n" then
+    if item == "y" then
-      self:notify "migration aborted"
+      migrate()
-      return
-    elseif to_sqlite then
-      if sqlite_module.can_use then
-        self.migrator:to_sqlite()
-      else
-        self:error "sqlite.lua is unavailable"
-        return
-      end
     else
-      self.migrator:to_v1()
+      self:notify "Migration aborted"
     end
-    self:notify "migration finished successfully"
   end)
 end