コミットを比較

...
31 コミット
作成者 SHA1 メッセージ 日付
Codex Night Worker 9c579e1613 WIP: preserve failed Codex job issue-414-euwzily9 2026-11-19 12:29:29 +09:00
みてるぞ 4c17d0ef06 #414 2026-11-18 22:55:54 +09:00
みてるぞ ef5a3f725e Merge remote-tracking branch 'origin/main' into feature/414 2026-11-17 04:07:45 +09:00
みてるぞとCodex Night Worker dda3d87073 外部タグの履歴管理方式を変更 (#445) (#480)
Co-authored-by: Codex Night Worker <codex-night@localhost>
Reviewed-on: #480
Co-authored-by: miteruzo <miteruzo@naver.com>
2026-11-15 16:18:27 +09:00
テツノニジカとCodex Night Worker e13a52db9a AI: #454 広場のタグ検索,スマホ版だとサゼストが途中で見切れる問題 (#456)
Refs #454

## Automated implementation (requires human review)
[SidebarComponent.tsx](frontend/src/components/layout/SidebarComponent.tsx) の1行を修正しました。`overflow-y-auto` を `md` 以上に限定し、スマホでサゼストがサイドバーの領域外で切られる原因を取り除きました。

差分を確認済みです。指定に従い、テスト・lint・ビルドは未実行です。実画面でのスマホ表示とPC表示の確認も未実施です。

## Automated review (not a guarantee of correctness)
```text
The change restricts sidebar scrolling to medium and larger viewports. No actionable regression was identified in the current diff.

```

## Verification
- Tests/builds/migrations were not authorized and were not invoked by the controller.
- Codex was instructed not to run them; its command transcript should be inspected.
- Do not merge before manual acceptance and required checks.

---------

Co-authored-by: Codex Night Worker <codex-night@localhost>
Reviewed-on: #456
Co-authored-by: テツノニジカ <969+codex-bot@noreply.localhost>
2026-11-14 17:09:38 +09:00
みてるぞ f05d8a4b3a 集計で独裁者以外がほとんど検索条件を指定できないバグ (#444) (#452)
Reviewed-on: #452
Co-authored-by: miteruzo <miteruzo@naver.com>
2026-11-09 00:03:26 +09:00
みてるぞ dbae864f24 広場のタグ検索がスマホで利かない問題 (#435) (#453)
Reviewed-on: #453
Co-authored-by: miteruzo <miteruzo@naver.com>
2026-11-08 14:52:38 +09:00
みてるぞ 8d00a6545e API の処理に時間がかかりすぎる問題 (#442) (#449)
Reviewed-on: #449
Co-authored-by: miteruzo <miteruzo@naver.com>
2026-11-07 19:01:20 +09:00
みてるぞ 971b2632b0 #414 2026-11-03 15:17:10 +09:00
みてるぞ ad04c03e14 #414 2026-11-03 13:37:09 +09:00
みてるぞ 2197a9192b #414 2026-11-03 12:22:15 +09:00
みてるぞ f8747a0f74 Merge pull request '集計機能追加 (#434)' (#441) from feature/434 into main
Reviewed-on: #441
2026-11-01 19:39:04 +09:00
みてるぞ f874d6e405 #434 2026-11-01 19:34:52 +09:00
みてるぞ 22e57564a5 #434 2026-11-01 18:18:54 +09:00
みてるぞ 29002d5393 #434 2026-11-01 04:22:26 +09:00
みてるぞ 8a4c1d1220 #434 2026-11-01 01:12:29 +09:00
みてるぞ 74c5242828 #434 2026-11-01 00:20:32 +09:00
みてるぞ 6afbab8eee #434 2026-10-31 23:30:45 +09:00
みてるぞ 14853a8c5f #434 2026-10-31 20:44:28 +09:00
みてるぞ 4f3990feb6 #434 2026-10-31 19:07:14 +09:00
みてるぞ 5b77285a6b #434 2026-10-31 14:25:31 +09:00
みてるぞ dba409c3e8 #434 2026-10-31 13:50:10 +09:00
みてるぞ 6a0f1ec819 #434 2026-10-31 04:08:05 +09:00
みてるぞ efbd640963 #434 2026-10-29 23:31:04 +09:00
みてるぞ 7fc2faeaf0 #434 2026-10-29 22:26:49 +09:00
みてるぞ 237331481f #434 2026-10-29 21:38:38 +09:00
みてるぞ e857a33ec3 #434 2026-10-29 19:49:05 +09:00
みてるぞ 9f92dbfb06 #434 2026-10-29 18:10:13 +09:00
みてるぞ 549a45e9d5 #434 2026-10-29 12:38:03 +09:00
みてるぞ 7090f63ab6 #434 2026-10-29 12:30:42 +09:00
みてるぞ 7536330ecf #434 2026-10-29 12:28:20 +09:00
167個のファイルの変更、10344行の追加、1524行の削除
-35
ファイルの表示
@@ -1,35 +0,0 @@
## 背景
なぜ必要か。
## 対象範囲
- backend:
- frontend:
- docs:
- migration:
## やること
- [ ]
## 受け入れ条件
- [ ]
## 実行すべき確認
- [ ] `cd backend && bundle exec rspec`
- [ ] `cd frontend && npm run build`
- [ ] `cd frontend && npm run lint`
## 禁止事項
- unrelated refactor はしない
- 既存 API response shape を壊さない
- 認証・認可・BAN を弱めない
## Codex への指示
この issue を読んで実装してください。
不明点があれば、実装前に調査結果と選択肢を提示してください。
+2
ファイルの表示
@@ -71,3 +71,5 @@ gem "rspec-rails", "~> 8.0", :groups => [:development, :test]
gem 'aws-sdk-s3', require: false
gem 'rails-i18n', '~> 8.0.0'
gem "json", "~> 2.19.9"
+142 -134
ファイルの表示
@@ -1,29 +1,29 @@
GEM
remote: https://rubygems.org/
specs:
actioncable (8.0.2)
actionpack (= 8.0.2)
activesupport (= 8.0.2)
actioncable (8.0.5.1)
actionpack (= 8.0.5.1)
activesupport (= 8.0.5.1)
nio4r (~> 2.0)
websocket-driver (>= 0.6.1)
zeitwerk (~> 2.6)
actionmailbox (8.0.2)
actionpack (= 8.0.2)
activejob (= 8.0.2)
activerecord (= 8.0.2)
activestorage (= 8.0.2)
activesupport (= 8.0.2)
actionmailbox (8.0.5.1)
actionpack (= 8.0.5.1)
activejob (= 8.0.5.1)
activerecord (= 8.0.5.1)
activestorage (= 8.0.5.1)
activesupport (= 8.0.5.1)
mail (>= 2.8.0)
actionmailer (8.0.2)
actionpack (= 8.0.2)
actionview (= 8.0.2)
activejob (= 8.0.2)
activesupport (= 8.0.2)
actionmailer (8.0.5.1)
actionpack (= 8.0.5.1)
actionview (= 8.0.5.1)
activejob (= 8.0.5.1)
activesupport (= 8.0.5.1)
mail (>= 2.8.0)
rails-dom-testing (~> 2.2)
actionpack (8.0.2)
actionview (= 8.0.2)
activesupport (= 8.0.2)
actionpack (8.0.5.1)
actionview (= 8.0.5.1)
activesupport (= 8.0.5.1)
nokogiri (>= 1.8.5)
rack (>= 2.2.4)
rack-session (>= 1.0.1)
@@ -31,35 +31,35 @@ GEM
rails-dom-testing (~> 2.2)
rails-html-sanitizer (~> 1.6)
useragent (~> 0.16)
actiontext (8.0.2)
actionpack (= 8.0.2)
activerecord (= 8.0.2)
activestorage (= 8.0.2)
activesupport (= 8.0.2)
actiontext (8.0.5.1)
actionpack (= 8.0.5.1)
activerecord (= 8.0.5.1)
activestorage (= 8.0.5.1)
activesupport (= 8.0.5.1)
globalid (>= 0.6.0)
nokogiri (>= 1.8.5)
actionview (8.0.2)
activesupport (= 8.0.2)
actionview (8.0.5.1)
activesupport (= 8.0.5.1)
builder (~> 3.1)
erubi (~> 1.11)
rails-dom-testing (~> 2.2)
rails-html-sanitizer (~> 1.6)
activejob (8.0.2)
activesupport (= 8.0.2)
activejob (8.0.5.1)
activesupport (= 8.0.5.1)
globalid (>= 0.3.6)
activemodel (8.0.2)
activesupport (= 8.0.2)
activerecord (8.0.2)
activemodel (= 8.0.2)
activesupport (= 8.0.2)
activemodel (8.0.5.1)
activesupport (= 8.0.5.1)
activerecord (8.0.5.1)
activemodel (= 8.0.5.1)
activesupport (= 8.0.5.1)
timeout (>= 0.4.0)
activestorage (8.0.2)
actionpack (= 8.0.2)
activejob (= 8.0.2)
activerecord (= 8.0.2)
activesupport (= 8.0.2)
activestorage (8.0.5.1)
actionpack (= 8.0.5.1)
activejob (= 8.0.5.1)
activerecord (= 8.0.5.1)
activesupport (= 8.0.5.1)
marcel (~> 1.0)
activesupport (8.0.2)
activesupport (8.0.5.1)
base64
benchmark (>= 0.3)
bigdecimal
@@ -92,22 +92,22 @@ GEM
aws-sigv4 (~> 1.5)
aws-sigv4 (1.12.1)
aws-eventstream (~> 1, >= 1.0.2)
base64 (0.2.0)
base64 (0.3.0)
bcrypt_pbkdf (1.1.1)
bcrypt_pbkdf (1.1.1-arm64-darwin)
bcrypt_pbkdf (1.1.1-x86_64-darwin)
benchmark (0.4.0)
bigdecimal (3.1.9)
benchmark (0.5.0)
bigdecimal (4.1.3)
bootsnap (1.18.6)
msgpack (~> 1.2)
brakeman (7.0.2)
racc
builder (3.3.0)
chronic (0.10.2)
concurrent-ruby (1.3.5)
connection_pool (2.5.3)
crass (1.0.6)
date (3.4.1)
concurrent-ruby (1.3.8)
connection_pool (3.0.2)
crass (1.0.7)
date (3.6.0)
diff-lcs (1.6.2)
discard (1.4.0)
activerecord (>= 4.2, < 9.0)
@@ -115,8 +115,9 @@ GEM
dotenv-rails (3.1.8)
dotenv (= 3.1.8)
railties (>= 6.1)
drb (2.2.1)
drb (2.2.3)
ed25519 (1.4.0)
erb (6.0.7)
erubi (1.13.1)
factory_bot (6.5.6)
activesupport (>= 6.1.0)
@@ -134,7 +135,7 @@ GEM
gemojione (4.3.3)
json
github-markup (4.0.2)
globalid (1.2.1)
globalid (1.4.0)
activesupport (>= 6.1)
gollum (6.1.0)
gemojione (~> 4.1)
@@ -166,19 +167,20 @@ GEM
gollum-rugged_adapter (3.0)
mime-types (~> 3.4)
rugged (~> 1.5)
i18n (1.14.7)
i18n (1.15.2)
concurrent-ruby (~> 1.0)
image_processing (1.14.0)
mini_magick (>= 4.9.5, < 6)
ruby-vips (>= 2.0.17, < 3)
io-console (0.8.0)
irb (1.15.2)
io-console (0.9.4)
irb (1.18.0)
pp (>= 0.6.0)
prism (>= 1.3.0)
rdoc (>= 4.0.0)
reline (>= 0.4.2)
jmespath (1.6.2)
json (2.12.0)
jwt (2.10.1)
json (2.19.9)
jwt (3.3.0)
base64
kamal (2.6.1)
activesupport (>= 7.0)
@@ -198,15 +200,16 @@ GEM
language_server-protocol (3.17.0.5)
lint_roller (1.1.0)
logger (1.7.0)
loofah (2.24.1)
loofah (2.25.2)
crass (~> 1.0.2)
nokogiri (>= 1.12.0)
mail (2.8.1)
mail (2.9.1)
logger
mini_mime (>= 0.1.1)
net-imap
net-pop
net-smtp
marcel (1.0.4)
marcel (1.2.1)
mime-types (3.7.0)
logger
mime-types-data (~> 3.2025, >= 3.2025.0507)
@@ -215,45 +218,46 @@ GEM
benchmark
logger
mini_mime (1.1.5)
minitest (5.25.5)
msgpack (1.8.0)
multi_json (1.15.0)
minitest (6.0.6)
drb (~> 2.0)
prism (~> 1.5)
msgpack (1.8.5)
multi_json (1.21.2)
mustache (1.1.1)
mustache-sinatra (2.0.0)
mustache (~> 1.0)
mustermann (3.0.3)
ruby2_keywords (~> 0.0.1)
mustermann (3.1.1)
mysql2 (0.5.6)
net-imap (0.5.8)
net-imap (0.6.8)
date
net-protocol
net-pop (0.1.2)
net-protocol
net-protocol (0.2.2)
net-protocol (0.4.0)
timeout
net-scp (4.1.0)
net-ssh (>= 2.6.5, < 8.0.0)
net-sftp (4.0.0)
net-ssh (>= 5.0.0, < 8.0.0)
net-smtp (0.5.1)
net-smtp (0.5.2)
net-protocol
net-ssh (7.3.0)
nio4r (2.7.4)
nokogiri (1.18.8-aarch64-linux-gnu)
nio4r (2.7.5)
nokogiri (1.19.4-aarch64-linux-gnu)
racc (~> 1.4)
nokogiri (1.18.8-aarch64-linux-musl)
nokogiri (1.19.4-aarch64-linux-musl)
racc (~> 1.4)
nokogiri (1.18.8-arm-linux-gnu)
nokogiri (1.19.4-arm-linux-gnu)
racc (~> 1.4)
nokogiri (1.18.8-arm-linux-musl)
nokogiri (1.19.4-arm-linux-musl)
racc (~> 1.4)
nokogiri (1.18.8-arm64-darwin)
nokogiri (1.19.4-arm64-darwin)
racc (~> 1.4)
nokogiri (1.18.8-x86_64-darwin)
nokogiri (1.19.4-x86_64-darwin)
racc (~> 1.4)
nokogiri (1.18.8-x86_64-linux-gnu)
nokogiri (1.19.4-x86_64-linux-gnu)
racc (~> 1.4)
nokogiri (1.18.8-x86_64-linux-musl)
nokogiri (1.19.4-x86_64-linux-musl)
racc (~> 1.4)
octicons (19.15.2)
ostruct (0.6.1)
@@ -261,70 +265,73 @@ GEM
parser (3.3.8.0)
ast (~> 2.4.1)
racc
pp (0.6.2)
pp (0.6.4)
prettyprint
prettyprint (0.2.0)
prism (1.4.0)
psych (5.2.6)
prettyprint (0.2.1)
prism (1.9.0)
psych (5.5.0)
date
stringio
puma (6.6.0)
nio4r (~> 2.0)
racc (1.8.1)
rack (3.1.14)
rack (3.2.7)
rack-cors (2.0.2)
rack (>= 2.0.0)
rack-protection (4.1.1)
rack-protection (4.2.1)
base64 (>= 0.1.0)
logger (>= 1.6.0)
rack (>= 3.0.0, < 4)
rack-session (2.1.1)
rack-session (2.1.2)
base64 (>= 0.1.0)
rack (>= 3.0.0)
rack-test (2.2.0)
rack-test (2.3.0)
rack (>= 1.3)
rackup (2.2.1)
rackup (2.3.1)
rack (>= 3)
rails (8.0.2)
actioncable (= 8.0.2)
actionmailbox (= 8.0.2)
actionmailer (= 8.0.2)
actionpack (= 8.0.2)
actiontext (= 8.0.2)
actionview (= 8.0.2)
activejob (= 8.0.2)
activemodel (= 8.0.2)
activerecord (= 8.0.2)
activestorage (= 8.0.2)
activesupport (= 8.0.2)
rails (8.0.5.1)
actioncable (= 8.0.5.1)
actionmailbox (= 8.0.5.1)
actionmailer (= 8.0.5.1)
actionpack (= 8.0.5.1)
actiontext (= 8.0.5.1)
actionview (= 8.0.5.1)
activejob (= 8.0.5.1)
activemodel (= 8.0.5.1)
activerecord (= 8.0.5.1)
activestorage (= 8.0.5.1)
activesupport (= 8.0.5.1)
bundler (>= 1.15.0)
railties (= 8.0.2)
rails-dom-testing (2.2.0)
railties (= 8.0.5.1)
rails-dom-testing (2.3.0)
activesupport (>= 5.0.0)
minitest
nokogiri (>= 1.6)
rails-html-sanitizer (1.6.2)
loofah (~> 2.21)
rails-html-sanitizer (1.7.1)
loofah (~> 2.25, >= 2.25.2)
nokogiri (>= 1.15.7, != 1.16.7, != 1.16.6, != 1.16.5, != 1.16.4, != 1.16.3, != 1.16.2, != 1.16.1, != 1.16.0.rc1, != 1.16.0)
rails-i18n (8.0.2)
i18n (>= 0.7, < 2)
railties (>= 8.0.0, < 9)
railties (8.0.2)
actionpack (= 8.0.2)
activesupport (= 8.0.2)
railties (8.0.5.1)
actionpack (= 8.0.5.1)
activesupport (= 8.0.5.1)
irb (~> 1.13)
rackup (>= 1.0.0)
rake (>= 12.2)
thor (~> 1.0, >= 1.2.2)
tsort (>= 0.2)
zeitwerk (~> 2.6)
rainbow (3.1.1)
rake (13.2.1)
rdoc (6.13.1)
rake (13.4.2)
rdoc (6.17.0)
erb
psych (>= 4.0.0)
regexp_parser (2.10.0)
reline (0.6.1)
tsort
regexp_parser (2.13.1)
reline (0.7.0)
io-console (~> 0.5)
rexml (3.4.1)
rexml (3.4.4)
rouge (3.30.0)
rspec-core (3.13.6)
rspec-support (~> 3.13.0)
@@ -345,20 +352,20 @@ GEM
rspec-support (3.13.6)
rss (0.3.1)
rexml
rubocop (1.75.6)
json (~> 2.3)
rubocop (1.92.0)
json (>= 2.3)
language_server-protocol (~> 3.17.0.2)
lint_roller (~> 1.1.0)
parallel (~> 1.10)
parallel (>= 1.10)
parser (>= 3.3.0.2)
rainbow (>= 2.2.2, < 4.0)
regexp_parser (>= 2.9.3, < 3.0)
rubocop-ast (>= 1.44.0, < 2.0)
regexp_parser (>= 2.11.3, < 3.0)
rubocop-ast (>= 1.49.0, < 2.0)
ruby-progressbar (~> 1.7)
unicode-display_width (>= 2.4.0, < 4.0)
rubocop-ast (1.44.1)
rubocop-ast (1.50.0)
parser (>= 3.3.7.2)
prism (~> 1.4)
prism (~> 1.7)
rubocop-performance (1.25.0)
lint_roller (~> 1.1)
rubocop (>= 1.75.0, < 2.0)
@@ -377,24 +384,23 @@ GEM
ruby-vips (2.2.3)
ffi (~> 1.12)
logger
ruby2_keywords (0.0.5)
rugged (1.9.0)
securerandom (0.4.1)
sinatra (4.1.1)
sinatra (4.2.1)
logger (>= 1.6.0)
mustermann (~> 3.0)
rack (>= 3.0.0, < 4)
rack-protection (= 4.1.1)
rack-protection (= 4.2.1)
rack-session (>= 2.0.0, < 3)
tilt (~> 2.0)
sinatra-contrib (4.1.1)
sinatra-contrib (4.2.1)
multi_json (>= 0.0.2)
mustermann (~> 3.0)
rack-protection (= 4.1.1)
sinatra (= 4.1.1)
rack-protection (= 4.2.1)
sinatra (= 4.2.1)
tilt (~> 2.0)
sprockets (4.2.2)
concurrent-ruby (~> 1.0)
sprockets (4.4.1)
concurrent-ruby (~> 1.1)
logger
rack (>= 2.2.4, < 4)
sprockets-helpers (1.4.0)
@@ -403,14 +409,14 @@ GEM
actionpack (>= 6.1)
activesupport (>= 6.1)
sprockets (>= 3.0.0)
sqlite3 (2.6.0-aarch64-linux-gnu)
sqlite3 (2.6.0-aarch64-linux-musl)
sqlite3 (2.6.0-arm-linux-gnu)
sqlite3 (2.6.0-arm-linux-musl)
sqlite3 (2.6.0-arm64-darwin)
sqlite3 (2.6.0-x86_64-darwin)
sqlite3 (2.6.0-x86_64-linux-gnu)
sqlite3 (2.6.0-x86_64-linux-musl)
sqlite3 (2.10.1-aarch64-linux-gnu)
sqlite3 (2.10.1-aarch64-linux-musl)
sqlite3 (2.10.1-arm-linux-gnu)
sqlite3 (2.10.1-arm-linux-musl)
sqlite3 (2.10.1-arm64-darwin)
sqlite3 (2.10.1-x86_64-darwin)
sqlite3 (2.10.1-x86_64-linux-gnu)
sqlite3 (2.10.1-x86_64-linux-musl)
sshkit (1.24.0)
base64
logger
@@ -418,18 +424,19 @@ GEM
net-sftp (>= 2.1.2)
net-ssh (>= 2.8.0)
ostruct
stringio (3.1.7)
stringio (3.2.0)
therubyrhino (2.1.2)
therubyrhino_jar (>= 1.7.4, < 1.7.9)
therubyrhino_jar (1.7.8)
thor (1.3.2)
thor (1.5.0)
thruster (0.1.13)
thruster (0.1.13-aarch64-linux)
thruster (0.1.13-arm64-darwin)
thruster (0.1.13-x86_64-darwin)
thruster (0.1.13-x86_64-linux)
tilt (2.6.0)
timeout (0.4.3)
tilt (2.9.0)
timeout (0.6.1)
tsort (0.2.0)
twitter-text (1.14.7)
unf (~> 0.1.0)
tzinfo (2.0.6)
@@ -440,16 +447,16 @@ GEM
unicode-display_width (3.1.4)
unicode-emoji (~> 4.0, >= 4.0.4)
unicode-emoji (4.0.4)
uri (1.0.3)
uri (1.1.2)
useragent (0.16.11)
webrick (1.9.1)
websocket-driver (0.7.7)
websocket-driver (0.8.2)
base64
websocket-extensions (>= 0.1.0)
websocket-extensions (0.1.5)
whenever (1.0.0)
chronic (>= 0.6.3)
zeitwerk (2.7.2)
zeitwerk (2.8.3)
PLATFORMS
aarch64-linux
@@ -473,6 +480,7 @@ DEPENDENCIES
factory_bot_rails
gollum
image_processing (~> 1.14)
json (~> 2.19.9)
jwt
kamal
mysql2 (~> 0.5.6)
+3
ファイルの表示
@@ -27,6 +27,9 @@ class ApplicationController < ActionController::API
private
def enforce_query_contract!
if request.query_parameters.keys.any? { _1 == 'tag_refs' || _1.start_with?('tag_refs.') }
return head :bad_request
end
pairs = URI.decode_www_form(request.query_string)
return head :bad_request if pairs.length > 1000 || pairs.map(&:first).uniq.length != pairs.length
return head :bad_request if request.query_parameters.values.any? { !_1.is_a?(String) }
+109 -9
ファイルの表示
@@ -12,6 +12,15 @@ class CountsExportsController < ApplicationController
render json: { error: 'export_worker_unavailable' }, status: :service_unavailable
rescue CountsExportDownload::Busy
head :too_many_requests
rescue CountsAxes::TooManyAxes
render json: { error: 'too_many_axes', max_axes: CountsAxes::MAX_AXES }, status: :bad_request
rescue CountsAxes::PrivateCondition
render json: { error: 'private_condition' }, status: :forbidden
rescue CountsTableQuery::TooManyColumns
render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS },
status: :unprocessable_entity
rescue ActiveRecord::StatementTimeout
render json: { error: 'counts_query_timeout' }, status: :service_unavailable
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
head :bad_request
end
@@ -24,6 +33,14 @@ class CountsExportsController < ApplicationController
job_state(cancel: true)
end
def start_download
result = CountsExportJobs.download_attempt(params[:token], session[:counts_export_owner])
return head :not_found unless result
response.headers['Cache-Control'] = 'private, no-store'
render json: result
end
def prepare
locale = resolve_locale!
payload = request.request_parameters.merge(request.query_parameters)
@@ -40,11 +57,25 @@ class CountsExportsController < ApplicationController
head :gateway_timeout
rescue CountsExportDownload::StorageFull, Errno::ENOSPC
head :insufficient_storage
rescue CountsAxes::TooManyAxes
render json: { error: 'too_many_axes', max_axes: CountsAxes::MAX_AXES }, status: :bad_request
rescue CountsAxes::PrivateCondition
render json: { error: 'private_condition' }, status: :forbidden
rescue CountsTableQuery::TooManyColumns
render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS },
status: :unprocessable_entity
rescue ActiveRecord::StatementTimeout
render json: { error: 'counts_query_timeout' }, status: :service_unavailable
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
head :bad_request
end
def download
if params[:job].present? || params[:attempt].present?
return head :method_not_allowed unless request.get?
return confirmed_download
end
file = CountsExportDownload.find(params[:token], session[:counts_export_owner])
return head :not_found unless file
@@ -55,15 +86,6 @@ class CountsExportsController < ApplicationController
disposition: 'attachment'
end
def job_state(cancel:)
result = CountsExportJobs.status(params[:token], session[:counts_export_owner], cancel:)
return head :not_found unless result
response.headers['Cache-Control'] = 'private, no-store'
render json: result
end
private :job_state
def index
locale = resolve_locale!
payload = request.request_parameters.except('authenticity_token')
@@ -79,7 +101,85 @@ class CountsExportsController < ApplicationController
filename: "counts-#{ exporter.resource }.#{ exporter.format }",
type: exporter.format == 'json' ? 'application/json' : 'text/csv',
disposition: 'attachment'
rescue CountsAxes::TooManyAxes
render json: { error: 'too_many_axes', max_axes: CountsAxes::MAX_AXES }, status: :bad_request
rescue CountsAxes::PrivateCondition
render json: { error: 'private_condition' }, status: :forbidden
rescue CountsTableQuery::TooManyColumns
render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS },
status: :unprocessable_entity
rescue ActiveRecord::StatementTimeout
render json: { error: 'counts_query_timeout' }, status: :service_unavailable
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
head :bad_request
end
private
def confirmed_download
owner = session[:counts_export_owner]
arguments = { file_token: params[:token], attempt: params[:attempt] }
if (!(arguments[:attempt].present?) ||
!(CountsExportJobs.download_attempt(params[:job], owner, **arguments)))
return head :not_found
end
file = CountsExportDownload.find(params[:token], owner)
unless file
CountsExportJobs.download_attempt(params[:job], owner, **arguments, status: 'unavailable')
return head :not_found
end
job = params[:job]
authority = HistoryAccess.user
first_check = true
state = lambda do |status|
if status == 'active'
if first_check
first_check = false
fresh = authority && User.find_by(id: authority.id)
next false if authority && (!fresh || fresh.banned?)
authority = fresh
HistoryAccess.set(user: authority) do
CountsExportDownload.find(arguments[:file_token], owner) &&
CountsExportJobs.stream_active?(job, owner, **arguments)
end
else
CountsExportJobs.stream_active?(job, owner, **arguments)
end
else
HistoryAccess.set(user: authority) do
CountsExportJobs.download_attempt(job, owner, **arguments, status:)
end
end
end
unless CountsExportJobs.download_attempt(job, owner, **arguments, status: 'streaming')
return head :conflict
end
response.headers['Cache-Control'] = 'private, no-store'
response.headers['Last-Modified'] = File.mtime(file.fetch('path')).httpdate
response.headers['X-Accel-Buffering'] = 'no'
send_file file.fetch('path'),
filename: "counts-#{ file.fetch('resource') }.#{ file.fetch('format') }",
type: file['format'] == 'json' ? 'application/json' : 'text/csv',
disposition: 'attachment'
# Keep Rails' bounded FileBody reader, but hide to_path from Rack::Sendfile:
# offloading would bypass the GET acknowledgement and cancellation checks.
stream = CountsExportDownload::ResponseBody.new(response.stream, &state)
self.response_body = stream
rescue IOError, SystemCallError
stream&.close
CountsExportJobs.download_attempt(params[:job], owner, **arguments, status: 'failed')
raise if response.committed?
head :service_unavailable
end
def job_state cancel:
result = CountsExportJobs.status(params[:token], session[:counts_export_owner], cancel:)
return head :not_found unless result
response.headers['Cache-Control'] = 'private, no-store'
render json: result
end
end
+9
ファイルの表示
@@ -18,9 +18,18 @@ class CountsTablesController < ApplicationController
def index
resolve_locale!
render json: CountsTableQuery.new(request.query_parameters).call
rescue CountsAxes::TooManyAxes
render json: { error: 'too_many_axes', max_axes: CountsAxes::MAX_AXES }, status: :bad_request
rescue CountsAxes::PrivateCondition
render json: { error: 'private_condition' }, status: :forbidden
rescue CountsTableQuery::TooManyColumns
render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS },
status: :unprocessable_entity
rescue CountsPivotTotals::TimedOut
Rails.logger.warn('counts_query_timeout: pivot_totals')
render json: { error: 'counts_query_timeout', stage: 'totals',
timeout_seconds: CountsTableQuery::SQL_TIMEOUT_SECONDS },
status: :service_unavailable
rescue ActiveRecord::StatementTimeout => error
Rails.logger.warn("counts_query_timeout: #{ error.class }")
render json: { error: 'counts_query_timeout' }, status: :service_unavailable
+6 -3
ファイルの表示
@@ -100,7 +100,8 @@ class MaterialsController < ApplicationController
material = nil
begin
Material.transaction do
VersionRecorder.record_operation!(
event_type: 'materials.create', created_by_user: current_user) do
tag = resolve_material_tag!(locale, tag_name_raw)
material = Material.new(tag:, url:,
created_by_user: current_user,
@@ -151,7 +152,8 @@ class MaterialsController < ApplicationController
uploaded_blob = build_uploaded_material_blob!(file, file_sha256)
begin
Material.transaction do
VersionRecorder.record_operation!(
event_type: 'materials.update', created_by_user: current_user) do
MaterialVersionRecorder.ensure_snapshot!(material, created_by_user: current_user)
tag = resolve_material_tag!(locale, tag_name_raw)
material.assign_attributes(tag:, url:, updated_by_user: current_user)
@@ -185,7 +187,8 @@ class MaterialsController < ApplicationController
material = Material.find_by(id: params[:id])
return head :not_found unless material
Material.transaction do
VersionRecorder.record_operation!(
event_type: 'materials.destroy', created_by_user: current_user) do
MaterialVersionRecorder.ensure_snapshot!(material, created_by_user: current_user)
material.discard!
MaterialVersionRecorder.record!(material:, event_type: :discard,
+2 -1
ファイルの表示
@@ -95,7 +95,8 @@ class NicoTagsController < ApplicationController
linked_tag_names = params[:tags].to_s.split
linked_tags = nil
ApplicationRecord.transaction do
VersionRecorder.record_operation!(
event_type: 'external_tags.update', created_by_user: current_user) do
linked_tags = Tag.normalise_tags!(locale, linked_tag_names,
with_tagme: false,
with_no_deerjikist: false,
+7 -12
ファイルの表示
@@ -14,8 +14,9 @@ class PostVersionsController < ApplicationController
offset = (page - 1) * limit
q = HistoryAccess.visible(version_scope)
q = HistoryAccess.visible(PostVersion.all)
q = q.where('post_versions.post_id = ?', post_id) if post_id
q = q.with_previous_version if tag_id || external_tag_id
if external_tag_id || (tag_id && !(Tag.exists?(id: tag_id)))
q = q.where('JSON_CONTAINS(post_versions.tags_json,' +
"JSON_OBJECT('external_tag_id', #{ external_tag_id || tag_id })) " +
@@ -28,9 +29,12 @@ class PostVersionsController < ApplicationController
count = q.except(:select, :order, :limit, :offset).count
versions = q.order(Arel.sql('post_versions.created_at DESC, post_versions.id DESC'))
ids = q.order(Arel.sql('post_versions.created_at DESC, post_versions.id DESC'))
.limit(limit)
.offset(offset)
.pluck('post_versions.id')
versions = version_scope.merge(q).where(id: ids)
.order(Arel.sql('post_versions.created_at DESC, post_versions.id DESC'))
render json: { versions: serialise_versions(versions, locale.language_code), count:, population_count: HistoryAccess.population_count(PostVersion) }
end
@@ -96,18 +100,9 @@ class PostVersionsController < ApplicationController
def external_tag_names_for snapshots
ids = snapshots.flatten.filter_map { _1['external_tag_id'] }.uniq
names = ExternalTag.where(id: ids).pluck(:id, :platform, :name).to_h { |id, platform, name|
ExternalTag.where(id: ids).pluck(:id, :platform, :name).to_h { |id, platform, name|
[id, "#{ platform }:#{ name }"]
}
missing_ids = ids - names.keys
HistoryAccess.visible(NicoTagVersion.all)
.where(tag_id: missing_ids)
.order(:tag_id, version_no: :desc)
.pluck(:tag_id, :name)
.each { |id, name| names[id] ||= name }
names
end
def snapshot_tag_literals snapshots, tag_names, external_tag_names
+62 -64
ファイルの表示
@@ -37,20 +37,10 @@ class PostsController < ApplicationController
offset = (page - 1) * limit
pt_max_sql =
HistoryAccess.visible(PostVersion.all)
.select('post_id, MAX(created_at) AS max_updated_at')
.group('post_id')
.to_sql
updated_at_all_sql =
'GREATEST(posts.updated_at,' +
'COALESCE(pt_max.max_updated_at, posts.updated_at))'
needs_history_join = order[0] == 'updated_at' || updated_between.any?(&:present?)
q =
filtered_posts(locale)
.joins("LEFT JOIN (#{ pt_max_sql }) pt_max ON pt_max.post_id = posts.id")
.reselect('posts.*', Arel.sql("#{ updated_at_all_sql } AS updated_at_all"))
.preload(:external_tags, :uploaded_user, :parents, :children,
post_tags: [:sections, { tag: [:deerjikists, :materials,
:tag_names, :tag_descriptions] }])
@@ -66,12 +56,25 @@ class PostsController < ApplicationController
end
q = q.where('posts.created_at >= ?', created_between[0]) if created_between[0]
q = q.where('posts.created_at <= ?', created_between[1]) if created_between[1]
count_scope = q
if needs_history_join
history_times = HistoryAccess.visible(PostVersion.all)
.select('post_id, MAX(created_at) AS latest_at').group(:post_id)
q = q.joins("LEFT JOIN (#{ history_times.to_sql }) history_times " +
'ON history_times.post_id = posts.id')
updated_at_all_sql = 'GREATEST(posts.updated_at, ' +
'COALESCE(history_times.latest_at, posts.updated_at))'
end
if updated_between[0]
q = q.where("#{ updated_at_all_sql } >= ?", updated_between[0])
end
if updated_between[1]
q = q.where("#{ updated_at_all_sql } <= ?", updated_between[1])
end
count_scope = q if updated_between.any?(&:present?)
if needs_history_join
q = q.select('posts.*', "#{ updated_at_all_sql } AS updated_at")
end
sort_sql =
case order[0]
@@ -88,12 +91,20 @@ class PostsController < ApplicationController
.limit(limit)
.offset(offset)
.to_a
latest_version_times =
if needs_history_join
{ }
else
HistoryAccess.visible(PostVersion.where(post_id: posts.map(&:id)))
.group(:post_id).maximum(:created_at)
end
q = q.except(:select, :order)
count_scope = count_scope.except(:select, :order)
render json: { posts: posts.map { |post|
updated_at = [post.updated_at, latest_version_times[post.id]].compact.max
PostRepr.base(post, language_code: locale.language_code)
.merge(updated_at: post.updated_at_all).tap do |json|
.merge('updated_at' => updated_at).tap do |json|
json['thumbnail'] =
if post.thumbnail.attached?
rails_storage_proxy_url(post.thumbnail, only_path: false)
@@ -101,7 +112,7 @@ class PostsController < ApplicationController
nil
end
end
}, count: q.group_values.present? ? q.count.size : q.count }
}, count: count_scope.group_values.present? ? count_scope.count.size : count_scope.count }
end
def random
@@ -335,7 +346,8 @@ class PostsController < ApplicationController
post = nil
conflict_json = nil
ApplicationRecord.transaction do
VersionRecorder.record_operation!(
event_type: 'posts.update', created_by_user: current_user) do
post = Post.lock.find(params[:id].to_i)
base_version = nil
@@ -409,52 +421,40 @@ class PostsController < ApplicationController
target = Locale.find_by(code:, deprecated_at: nil)
return head :not_found unless target
literals = params[:tags].to_s.split
references = tag_filter_references(literals)
names = literals.each_with_index.map do |literal, index|
reference = references&.[](index)
(reference && reference['negative'] == false) ? literal : literal.sub(/\Anot:/i, '')
end
internal = references ? {} : TagName.where(
language_code: source.language_code, name: names).pluck(:name, :tag_id).to_h
canonical = references ? {} : TagName.where(
language_code: source.language_code, primary_flg: true,
tag_id: internal.values).pluck(:tag_id, :name).to_h
external_names = names.map { canonical[internal[_1]] || _1 }
external = references ? {} : ExternalTag.where(
"CONCAT(platform, ':', name) IN (?)", external_names).pluck(
Arel.sql("CONCAT(platform, ':', name)"), :id).to_h
ids = references ? references.filter_map { _1['tag_id'] } : internal.values
translated = TagName.where(language_code: target.language_code,
primary_flg: true, tag_id: ids).pluck(:tag_id, :name).to_h
tokens = params[:tags].to_s.split
literals = PostTagSearch.literals(source, tokens.join(' '))
source_ids = PostTagSearch.references_for(source, literals)
ids = source_ids.values.flat_map(&:first).uniq
names = TagName.where(language_code: target.language_code,
primary_flg: true, tag_id: ids).pluck(:tag_id, :name).to_h
filters = literals.map do |literal|
internal, = source_ids.fetch(literal[:name])
name = internal.empty? ? literal[:name] : names[internal.first]
return head :unprocessable_entity unless name && internal.length <= 1
filters = literals.each_with_index.map do |literal, index|
name = names[index]
reference = if references
references[index].except('name')
else
matches = {}
matches['tag_id'] = internal[name] if internal[name]
external_id = external[external_names[index]]
matches['external_tag_id'] = external_id if external_id
matches.empty? ? { 'missing' => true } : matches
end
name = translated[reference['tag_id']] || name
negative = reference.fetch('negative', literal.downcase.start_with?('not:'))
prefix = negative ? literal[/\Anot:/i] || 'not:' : ''
reference.merge('name' => "#{ prefix }#{ name }", 'negative' => negative)
literal[:negative] ? "not:#{ name }" : name
end
render json: { references: filters }
translated = PostTagSearch.literals(target, filters.join(' '))
target_ids = PostTagSearch.references_for(target, translated)
literals.zip(translated).each do |literal, translation|
unless (source_ids.fetch(literal[:name]).map(&:sort) ==
target_ids.fetch(translation[:name]).map(&:sort))
return head :unprocessable_entity
end
end
index = 0
filters = tokens.map do |token|
next token if token.downcase == 'type:or'
value = filters[index]
index += 1
value
end
render json: { tags: filters.join(' ') }
end
private
def tag_filter_references literals
PostTagSearch.parse_references(params.to_unsafe_h, literals)
rescue ArgumentError, JSON::ParserError, TypeError
raise ActionController::BadRequest
end
def filtered_posts locale
tag_names = params[:tags].to_s.split
match_type = params[:match]
@@ -466,19 +466,17 @@ class PostsController < ApplicationController
end
def filter_posts_by_tags locale, tag_names, match_type
references = tag_filter_references(tag_names)
literals = PostTagSearch.literals(locale, tag_names.join(' '), references:)
literals = PostTagSearch.literals(locale, tag_names.join(' '))
return Post.all if literals.empty?
if match_type == 'any'
literals.reduce(Post.none) do |posts, literal|
posts.or(tag_literal_relation(locale, literal[:name], negative: literal[:negative],
reference: literal[:reference]))
posts.or(tag_literal_relation(locale, literal[:name], negative: literal[:negative]))
end
else
literals.reduce(Post.all) do |posts, literal|
ids = tagged_post_ids_for(locale, literal[:name], reference: literal[:reference])
ids = tagged_post_ids_for(locale, literal[:name])
if literal[:negative]
posts.where.not(id: ids)
else
@@ -488,8 +486,8 @@ class PostsController < ApplicationController
end
end
def tag_literal_relation locale, name, negative:, reference: nil
ids = tagged_post_ids_for(locale, name, reference:)
def tag_literal_relation locale, name, negative:
ids = tagged_post_ids_for(locale, name)
if negative
Post.where.not(id: ids)
else
@@ -497,8 +495,8 @@ class PostsController < ApplicationController
end
end
def tagged_post_ids_for locale, name, reference: nil
internal, external = PostTagSearch.references(locale, { name:, reference: })
def tagged_post_ids_for locale, name
internal, external = PostTagSearch.references(locale, { name: })
Post.where(id: PostTag.where(tag_id: Array(internal)).select(:post_id))
.or(Post.where(id: PostExternalTag.where(external_tag_id: Array(external))
.select(:post_id))).select(:id)
+4 -2
ファイルの表示
@@ -11,7 +11,8 @@ class TagChildrenController < ApplicationController
parent = Tag.find(parent_id)
child = Tag.find(child_id)
ApplicationRecord.transaction do
VersionRecorder.record_operation!(
event_type: 'tags.update', created_by_user: current_user) do
TagVersioning.ensure_snapshot!(child, created_by_user: current_user)
TagImplication.find_or_create_by!(parent_tag: parent, tag: child)
@@ -33,7 +34,8 @@ class TagChildrenController < ApplicationController
parent = Tag.find(parent_id)
child = Tag.find(child_id)
ApplicationRecord.transaction do
VersionRecorder.record_operation!(
event_type: 'tags.update', created_by_user: current_user) do
TagVersioning.ensure_snapshot!(child, created_by_user: current_user)
TagImplication.find_by(parent_tag: parent, tag: child)&.destroy!
+4 -2
ファイルの表示
@@ -467,7 +467,8 @@ class TagsController < ApplicationController
return render_unprocessable_entity 'ニコタグは変更できません.', field: :category
end
ApplicationRecord.transaction do
VersionRecorder.record_operation!(
event_type: 'tags.update', created_by_user: current_user) do
TagVersioning.ensure_snapshot!(tag, created_by_user: current_user,
language_code: locale.language_code)
@@ -518,7 +519,8 @@ class TagsController < ApplicationController
return render_unprocessable_entity 'ニコタグは変更できません.', field: :category
end
ApplicationRecord.transaction do
VersionRecorder.record_operation!(
event_type: 'tags.update', created_by_user: current_user) do
TagVersioning.ensure_snapshot!(tag, created_by_user: current_user,
language_code: locale.language_code)
+3 -1
ファイルの表示
@@ -246,7 +246,9 @@ class WikiPagesController < ApplicationController
base_revision_id = params[:base_revision_id].presence
ApplicationRecord.transaction do
VersionRecorder.record_operation!(
event_type: 'tag_descriptions.update',
created_by_user: current_user, reason: params[:message].presence) do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
tag.lock!
old_title = tag_name.name
+10
ファイルの表示
@@ -0,0 +1,10 @@
class Event < ApplicationRecord
has_many :post_versions
has_many :tag_versions
has_many :nico_tag_versions
has_many :material_versions
has_many :tag_description_versions
has_many :tag_group_versions
belongs_to :created_by_user, class_name: 'User', optional: true
end
+19 -6
ファイルの表示
@@ -1,8 +1,11 @@
class ExternalTag < ApplicationRecord
before_create { SharedResourceIdAllocator.assign!(self) }
# Must run before dependent association callbacks.
before_destroy :prevent_destroy
enum :platform, nico: 'nico'
validates :platform, presence: true, inclusion: { in: ExternalTag.platforms.keys }
validate :immutable_identity, on: :update
has_many :post_external_tags, dependent: :delete_all
has_many :posts, through: :post_external_tags
@@ -16,11 +19,21 @@ class ExternalTag < ApplicationRecord
has_many :linked_tags, through: :nico_tag_relations, source: :tag
def snapshot_linked_tag_names language_code = 'ja'
linked_tags
.joins(:tag_names)
.where(tag_names: { language_code:, primary_flg: true })
.order('tag_names.name')
.pluck('tag_names.name')
def snapshot_linked_tag_ids = linked_tags.distinct.order(:id).pluck(:id)
def delete
raise ActiveRecord::ReadOnlyRecord, '外部タグは削除できません.'
end
private
def immutable_identity
errors.add(:platform, 'は変更できません.') if will_save_change_to_platform?
errors.add(:name, 'は変更できません.') if will_save_change_to_name?
end
def prevent_destroy
errors.add(:base, '外部タグは削除できません.')
throw(:abort)
end
end
+34 -6
ファイルの表示
@@ -4,6 +4,7 @@ require 'digest'
class HistoryAccess < ActiveSupport::CurrentAttributes
attribute :user
class Forbidden < StandardError; end
ARTIFACT_POLICY = 'counts-private-predicates-4'.freeze
KEYS = { 'post_versions' => ['post_id'], 'tag_versions' => ['tag_id'],
'nico_tag_versions' => ['tag_id'],
@@ -18,10 +19,12 @@ class HistoryAccess < ActiveSupport::CurrentAttributes
end
def self.authority
{ 'policy_revision' => revision, 'admin' => admin?, 'actor_id' => user&.id }
{ 'artifact_policy' => ARTIFACT_POLICY, 'policy_revision' => revision,
'admin' => admin?, 'actor_id' => user&.id }
end
def self.authorised_artifact?(metadata)
return false unless metadata['artifact_policy'] == ARTIFACT_POLICY
return false unless metadata['policy_revision'].is_a?(String)
if metadata['admin']
admin? && user.id == metadata['actor_id']
@@ -56,20 +59,41 @@ class HistoryAccess < ActiveSupport::CurrentAttributes
end
def self.json_reference(table, key, document)
"NOT EXISTS (SELECT 1 FROM JSON_TABLE(COALESCE(#{ document }, JSON_ARRAY()), " +
# A scalar lookup avoids MySQL's semijoin rewrite losing the JSON row correlation.
"(SELECT 1 FROM JSON_TABLE(COALESCE(#{ document }, JSON_ARRAY()), " +
"'$[*]' COLUMNS (reference_id BIGINT PATH '$' NULL ON EMPTY)) reference_entry " +
"JOIN (#{ discarded_keys(table, key) }) blocked " +
"ON reference_entry.reference_id = blocked.#{ key })"
"ON reference_entry.reference_id = blocked.#{ key } LIMIT 1) IS NULL"
end
def self.list_reference(table, key, document)
# Preserve FIND_IN_SET's exact token semantics, including empty tokens.
array = "CONCAT('[', REPLACE(JSON_QUOTE(REPLACE(COALESCE(#{ document }, ''), " +
"' ', ',')), ',', '\",\"'), ']')"
"NOT EXISTS (SELECT 1 FROM JSON_TABLE(#{ array }, '$[*]' " +
"(SELECT 1 FROM JSON_TABLE(#{ array }, '$[*]' " +
"COLUMNS (reference_id VARCHAR(255) PATH '$')) reference_entry " +
"JOIN (#{ discarded_keys(table, key) }) blocked " +
"ON reference_entry.reference_id = CAST(blocked.#{ key } AS CHAR))"
"ON reference_entry.reference_id = CAST(blocked.#{ key } AS CHAR) LIMIT 1) IS NULL"
end
def self.snapshot_reference(model, row, external: false)
key = external ? 'external_tag_id' : 'tag_id'
table = model.table_name
source = "JSON_TABLE(COALESCE(#{ row }.tags_json, JSON_ARRAY()), '$[*]' COLUMNS (" +
"tag_id BIGINT PATH '$.#{ key }', version_no INT PATH '$.version_no')) tag_reference"
match = 'reference_version.tag_id = tag_reference.tag_id'
if external
match += " AND reference_version.created_at <= #{ row }.created_at"
latest = "AND NOT EXISTS (SELECT 1 FROM #{ table } later_reference " +
'WHERE later_reference.tag_id = reference_version.tag_id ' +
"AND later_reference.created_at <= #{ row }.created_at " +
'AND later_reference.version_no > reference_version.version_no)'
else
match += ' AND reference_version.version_no = tag_reference.version_no'
latest = ''
end
"(SELECT 1 FROM #{ source } JOIN #{ table } reference_version ON #{ match } " +
"WHERE NOT (#{ sql(model, 'reference_version') }) #{ latest } LIMIT 1) IS NULL"
end
def self.reference_sql(table, row)
@@ -78,7 +102,11 @@ class HistoryAccess < ActiveSupport::CurrentAttributes
[['tag_versions', 'tag_id', 'tag_id'],
['nico_tag_versions', 'tag_id', 'external_tag_id']].map do |target, key, field|
json_reference(target, key, "JSON_EXTRACT(#{ row }.tags_json, '$[*].#{ field }')")
end + [list_reference('post_versions', 'post_id', "#{ row }.parent_post_ids")]
end + [list_reference('post_versions', 'post_id', "#{ row }.parent_post_ids"),
snapshot_reference(TagVersion, row),
snapshot_reference(NicoTagVersion, row, external: true)]
when 'nico_tag_versions'
[list_reference('tag_versions', 'tag_id', "#{ row }.linked_tag_ids")]
when 'tag_versions'
[list_reference('tag_versions', 'tag_id', "#{ row }.parent_tag_ids")]
when 'tag_description_versions'
+3 -1
ファイルの表示
@@ -23,7 +23,9 @@ class Locale < ApplicationRecord
end
def save_with_tag_names!
completed = self.class.transaction(requires_new: true) do
event_type = new_record? ? 'locales.create' : 'locales.restore'
completed = VersionRecorder.record_operation!(
event_type:, created_by_user: nil, savepoint_flg: true) do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
saved = yield
raise ActiveRecord::RecordNotSaved, 'Locale save was halted' unless saved
+1 -3
ファイルの表示
@@ -11,7 +11,5 @@ class MaterialVersion < ApplicationRecord
belongs_to :parent, class_name: 'Material', optional: true
belongs_to :updated_by_user, class_name: 'User', optional: true
def export_paths_hash
export_paths_json || {}
end
def export_paths_hash = export_paths_json || { }
end
+6 -1
ファイルの表示
@@ -3,5 +3,10 @@ class NicoTagVersion < ApplicationRecord
belongs_to :external_tag, foreign_key: :tag_id, inverse_of: :nico_tag_versions
validates :name, presence: true
validates :event_type, inclusion: { in: ['create', 'update'] }
def name
tag = external_tag
"#{ tag.platform }:#{ tag.name }" if tag
end
end
+2 -1
ファイルの表示
@@ -290,7 +290,8 @@ class Tag < ApplicationRecord
affected_post_ids = Set.new
Tag.transaction do
VersionRecorder.record_operation!(
event_type: 'tags.update', created_by_user:) do
TagVersioning.ensure_snapshot!(target_tag, created_by_user:)
Array(source_tags).compact.uniq.each do |source_tag|
+28 -25
ファイルの表示
@@ -15,37 +15,40 @@ class TagNameSanitisationRule < ApplicationRecord
end
def apply!
Language.find_each do |language|
TagName.where(language:).find_each do |candidate|
TagName.transaction do
tn = TagName.find_by(id: candidate.id)
next unless tn
VersionRecorder.with_event(
event_type: 'tag_name_sanitisation_rules.apply', created_by_user: nil) do
Language.find_each do |language|
TagName.where(language:).find_each do |candidate|
TagName.transaction do
tn = TagName.find_by(id: candidate.id)
next unless tn
name = sanitise(tn.name)
next if name == tn.name
name = sanitise(tn.name)
next if name == tn.name
existing_tn = TagName.find_by(language:, name:)
source_tag = tn.tag
target_tag = existing_tn ? existing_tn.tag : source_tag
[source_tag, target_tag].uniq(&:id).sort_by(&:id).each(&:lock!)
TagVersioning.ensure_snapshot!(source_tag, created_by_user: nil)
existing_tn = TagName.find_by(language:, name:)
source_tag = tn.tag
target_tag = existing_tn ? existing_tn.tag : source_tag
[source_tag, target_tag].uniq(&:id).sort_by(&:id).each(&:lock!)
TagVersioning.ensure_snapshot!(source_tag, created_by_user: nil)
if existing_tn && target_tag != source_tag
Tag.merge_tags!(target_tag, source_tag)
tn.reload.destroy!
elsif existing_tn
if tn.primary?
existing_tn.destroy!
tn.update_columns(name:, updated_at: Time.current)
if existing_tn && target_tag != source_tag
Tag.merge_tags!(target_tag, source_tag)
tn.reload.destroy!
elsif existing_tn
if tn.primary?
existing_tn.destroy!
tn.update_columns(name:, updated_at: Time.current)
else
tn.destroy!
end
else
tn.destroy!
# TagName 側の自動サニタイズを回避
tn.update_columns(name:, updated_at: Time.current)
end
else
# TagName 側の自動サニタイズを回避
tn.update_columns(name:, updated_at: Time.current)
end
TagVersioning.record!(target_tag, event_type: :update, created_by_user: nil)
TagVersioning.record!(target_tag, event_type: :update, created_by_user: nil)
end
end
end
end
+2
ファイルの表示
@@ -19,6 +19,8 @@ module VersionRecord
belongs_to :created_by_user, class_name: 'User', optional: true
belongs_to :event
enum :event_type, event_type_map, prefix: true, validate: true
validates :version_no, presence: true, numericality: { only_integer: true, greater_than: 0 }
+4 -1
ファイルの表示
@@ -139,6 +139,8 @@ class CountsVersionRepr
result = {}
table = model.table_name
comparison = @inclusive ? '<=' : '<'
name = model == NicoTagVersion ?
CountsResourceDefinition.new('nico_tags').expression('name', table) : 'name'
requests.each_slice(100) do |batch|
queries = batch.map do |id, point|
scope = HistoryAccess.visible(model.where(tag_id: id))
@@ -148,7 +150,8 @@ class CountsVersionRepr
"AND newer.version_no > #{ table }.version_no)", point)
quoted = model.connection.quote(point.iso8601(6))
names = model == TagVersion ? 'names' : 'JSON_OBJECT() AS names'
scope.select("tag_id, name, #{ names }, #{ quoted } AS reference_point").to_sql
scope.select("tag_id, #{ name } AS name, #{ names }, " +
"#{ quoted } AS reference_point").to_sql
end
next if queries.empty?
+85 -3
ファイルの表示
@@ -1,6 +1,41 @@
class CountsAxes
MAX_AXES = 8
MASKED_TEXT = '********'.freeze
MASKED_VALUE = '__counts_private__'.freeze
PUBLIC_FIELDS = ['resource_id', 'version_no', 'event_type', 'user',
'created_at'].freeze
class TooManyAxes < ArgumentError; end
class PrivateCondition < ArgumentError; end
GRAINS = ['year', 'quarter', 'month', 'week', 'day', 'hour'].freeze
def self.private_field?(field)
!PUBLIC_FIELDS.include?(field) && field != 'user.id'
end
def self.visible_ids_sql(definition)
"SELECT counts_visible.id FROM #{ definition.table } counts_visible WHERE " +
HistoryAccess.sql(definition.model, 'counts_visible')
end
def self.visibility_sql(definition, field)
return 'TRUE' if HistoryAccess.admin?
source = visible_ids_sql(definition)
visible = "v.id IN (#{ source })"
return visible unless field == 'changed_field'
"(#{ visible }) AND (previous.id IS NULL OR previous.id IN (#{ source }))"
end
def self.masked_sql(expression, definition, field)
return expression if HistoryAccess.admin? || !private_field?(field)
visible = visibility_sql(definition, field)
quoted = ApplicationRecord.connection.quote(MASKED_VALUE)
"CASE WHEN #{ visible } THEN #{ expression } ELSE #{ quoted } END"
end
def self.time_expression(expression, grain, timezone)
raise ArgumentError unless GRAINS.include?(grain)
@@ -36,13 +71,14 @@ class CountsAxes
@rows = parse(population.params.fetch('rows', ''))
@columns = parse(population.params.fetch('cols', ''))
@all = rows + columns
raise ArgumentError if all.length > 8 || all.map { _1['field'] }.uniq.length != all.length
raise TooManyAxes if all.length > MAX_AXES
raise ArgumentError if all.map { _1['field'] }.uniq.length != all.length
end
def expressions
@expressions ||= all.map do |axis|
field = axis.fetch('field')
case field
expression = case field
when 'tags'
"CASE WHEN tag_entry.tag_id IS NOT NULL " +
"THEN CONCAT('internal:', tag_entry.tag_id) " +
@@ -53,14 +89,53 @@ class CountsAxes
@definition.fields[field] == :datetime ? self.class.time_expression(
expression, axis.fetch('grain', 'month'), @population.timezone) : expression
end
if !HistoryAccess.admin? && self.class.private_field?(field)
expression = self.class.masked_sql(
"CAST((#{ expression }) AS CHAR)", @definition, field)
end
expression
end
end
def grouped_sql(scope, indexes = all.each_index.to_a)
keys = indexes.map { "a#{ _1 }" }
selected = indexes.map { "#{ expressions[_1] } AS a#{ _1 }" }
identities = ['v.id']
if rows.any? { _1['field'] == 'tags' }
identities << self.class.masked_sql(
"COALESCE(CONCAT('internal:', tag_entry.tag_id), " +
"CONCAT('external:', tag_entry.external_tag_id))", @definition, 'tags')
if rows.any? { _1['field'] == 'changed_field' }
identities << self.class.masked_sql('changed.field_key', @definition, 'changed_field')
end
end
members = identities.each_index.map { "counts_member_#{ _1 }" }
selected += identities.each_with_index.map { |expression, index|
"#{ expression } AS #{ members[index] }"
}
# Evaluate correlated visibility subqueries before grouping. LIMIT prevents
# derived-table merging, so ONLY_FULL_GROUP_BY sees ordinary projected columns.
projection = scope.except(:select, :order).select(Arel.sql(selected.join(', ')))
.limit(18_446_744_073_709_551_615).to_sql
fields = keys + ["COUNT(DISTINCT #{ members.join(', ') }) AS count"]
grouping = keys.empty? ? '' : " GROUP BY #{ keys.join(', ') }"
"SELECT #{ fields.join(', ') } FROM (#{ projection }) counts_values#{ grouping }"
end
def joins(fields = all.map { _1['field'] })
result = ''
if fields.include?('tags')
document = 'CAST(v.tags_json AS JSON)'
unless HistoryAccess.admin?
visible = self.class.visible_ids_sql(@definition)
result += " LEFT JOIN (#{ visible }) counts_tags_visible " +
'ON counts_tags_visible.id = v.id '
document = "CASE WHEN counts_tags_visible.id IS NOT NULL " +
"THEN #{ document } ELSE JSON_ARRAY(" +
"JSON_OBJECT('tag_id', 0)) END"
end
result += <<~SQL.squish
JOIN JSON_TABLE(CAST(v.tags_json AS JSON), '$[*]' COLUMNS (
JOIN JSON_TABLE(#{ document }, '$[*]' COLUMNS (
tag_id BIGINT PATH '$.tag_id' NULL ON EMPTY,
external_tag_id BIGINT PATH '$.external_tag_id' NULL ON EMPTY)) tag_entry
@@ -73,6 +148,13 @@ class CountsAxes
condition = @definition.changes.map do |field|
"(changed.field_key = '#{ field }' AND #{ @population.changed_sql(field) })"
end.join(' OR ')
unless HistoryAccess.admin?
visible = self.class.visibility_sql(@definition, 'changed_field')
marker = ApplicationRecord.connection.quote(MASKED_VALUE)
fields += " UNION ALL SELECT #{ marker } AS field_key"
condition = "((#{ visible }) AND (#{ condition })) OR " +
"(NOT (#{ visible }) AND changed.field_key = #{ marker })"
end
result += " JOIN (#{ fields }) changed ON (#{ condition })"
end
result
+7 -6
ファイルの表示
@@ -28,30 +28,31 @@ class CountsCandidates
[external, 'external', 'NULL', 'JSON_OBJECT()']]
sources = sources.map do |population, kind, category, names|
scope = population.scope.except(:select)
name = population.definition.expression('name')
if @params['query'].present?
pattern = "%#{ ApplicationRecord.sanitize_sql_like(@params['query']) }%"
scope = scope.where("(v.name LIKE ? OR CAST(v.tag_id AS CHAR) = ? OR EXISTS (" +
scope = scope.where("(#{ name } LIKE ? OR CAST(v.tag_id AS CHAR) = ? OR EXISTS (" +
"SELECT 1 FROM JSON_TABLE(COALESCE(JSON_EXTRACT(#{ names }, '$**.name'), " +
"JSON_ARRAY()), '$[*]' COLUMNS (name TEXT PATH '$')) candidate_name " +
'WHERE candidate_name.name LIKE ?))', pattern, @params['query'], pattern)
end
scope.select(Arel.sql(
"CONCAT('#{ kind }:', v.tag_id) AS value, v.tag_id AS id, " +
"v.name AS name, #{ names } AS names, #{ category } AS category, '#{ kind }' AS kind"))
"#{ name } AS name, #{ names } AS names, " +
"#{ category } AS category, '#{ kind }' AS kind"))
.to_sql
end
clauses = []
binds = []
if @params['category'].present?
clauses << 'category = ?'
binds << @params['category']
clauses << ApplicationRecord.sanitize_sql_array(['category = ?', @params['category']])
end
where = clauses.empty? ? '' : "WHERE #{ clauses.join(' AND ') }"
sql = <<~SQL
SELECT /*+ MAX_EXECUTION_TIME(10000) */ * FROM (#{ sources.join(' UNION ALL ') }) choices
#{ where } ORDER BY kind, id LIMIT #{ LIMIT + 1 } OFFSET #{ (@page - 1) * LIMIT }
SQL
rows = connection.select_all(ApplicationRecord.sanitize_sql_array([sql, *binds])).to_a
# Sources already contain quoted search terms. Do not reinterpret their % or ? as binds.
rows = connection.select_all(sql).to_a
rows.each { _1['names'] = JSON.parse(_1['names']) if _1['names'].is_a?(String) }
{ rows: rows.first(LIMIT), page: @page, more: rows.length > LIMIT,
execution_at: internal.at.iso8601(6) }
+51 -21
ファイルの表示
@@ -82,6 +82,34 @@ class CountsCondition
def apply(scope, entry)
raise ArgumentError unless entry.is_a?(Hash)
if entry['operator'] == 'axis_eq'
raise ArgumentError unless (entry.keys - ['field', 'operator', 'grain', 'value']).empty?
value = entry['value']
unless (value.nil? || value.is_a?(String) || value.is_a?(Numeric) ||
[true, false].include?(value))
raise ArgumentError
end
raise ArgumentError if value.is_a?(String) && value.bytesize > 8192
if (!HistoryAccess.admin? && CountsAxes.private_field?(entry['field']) &&
(value == CountsAxes::MASKED_VALUE ||
(value == CountsAxes::MASKED_TEXT &&
(@definition.fields[entry['field']] != :string ||
['tags', 'changed_field'].include?(entry['field'])))))
field = entry.fetch('field')
raise ArgumentError unless @definition.axes(@population.history).include?(field)
if entry.key?('grain')
unless (@definition.fields[field] == :datetime &&
CountsAxes::GRAINS.include?(entry['grain']))
raise ArgumentError
end
end
return scope.none
end
scope = @population.condition_scope(scope, entry.fetch('field'))
return @population.axis_condition(scope, entry)
end
allowed = ['field', 'operator', 'value', 'end', 'blank', 'input', 'values']
raise ArgumentError if (entry.keys - allowed).any?
@@ -101,6 +129,13 @@ class CountsCondition
source, expression = source_for(field[:key])
type = field[:type]
# Preserve typed input compatibility; display markers are never real typed values.
if (!HistoryAccess.admin? && CountsAxes.private_field?(field[:key]) &&
entry['value'] == CountsAxes::MASKED_TEXT && type != :string && !entry['blank'])
return scope.none unless entry['operator'] == 'ne'
return @population.condition_scope(scope, field[:key])
end
blank = blank_sql(expression, type)
if entry['blank']
sql = if source && field[:collection]
@@ -109,6 +144,7 @@ class CountsCondition
"NOT EXISTS (SELECT 1 FROM #{ source } WHERE NOT (#{ blank }))"
else blank
end
scope = @population.condition_scope(scope, field[:key])
return scope.where(Arel.sql(sql))
end
if ['in', 'not_in', 'any', 'all', 'none'].include?(entry['operator'])
@@ -121,6 +157,7 @@ class CountsCondition
validate_enum!(field[:key], parsed.first) if type == :enum
operator = entry['operator']
if type == :interval
scope = @population.condition_scope(scope, field[:key])
return interval(scope, source, expression, parsed)
end
clauses = []
@@ -148,28 +185,10 @@ class CountsCondition
elsif operator == 'ne'
predicate = "(NOT (#{ predicate }) OR #{ blank })"
end
scope = @population.condition_scope(scope, field[:key])
scope.where(predicate, *binds)
end
def tag_name_match(value)
source = post_tag_source
bound = @population.history ? 'v.created_at' : @connection.quote(point)
source += ' LEFT JOIN nico_tag_versions external_snapshot ON external_snapshot.tag_id = ' +
scalar('tag_snapshot.item', 'external_tag_id', :integer) +
" AND external_snapshot.created_at <= #{ bound } AND " +
@population.access_sql(NicoTagVersion, 'external_snapshot') +
' AND NOT EXISTS (SELECT 1 FROM nico_tag_versions newer_external ' +
'WHERE newer_external.tag_id = external_snapshot.tag_id ' +
"AND newer_external.created_at <= #{ bound } " +
'AND newer_external.version_no > external_snapshot.version_no)'
document = "JSON_MERGE_PRESERVE(JSON_ARRAY(#{ scalar('tag_snapshot.item', 'name') }, " +
"referenced_tag.name, CONCAT('nico:', external_snapshot.name)), " +
"COALESCE(JSON_EXTRACT(referenced_tag.names, '$**.name'), JSON_ARRAY()))"
source += ' JOIN ' + json_table(document, '$[*]', 'search_name')
@definition.model.sanitize_sql_array([
"EXISTS (SELECT 1 FROM #{ source } WHERE JSON_UNQUOTE(search_name.item) = ?)", value])
end
private
def set_condition(scope, entry, field)
@@ -177,6 +196,13 @@ class CountsCondition
values = entry['values']
raise ArgumentError unless values.is_a?(Array) && values.length <= 100
if (!HistoryAccess.admin? && CountsAxes.private_field?(field[:key]) &&
values.include?(CountsAxes::MASKED_TEXT) &&
(field[:type] != :string || field[:set_type] == 'tag_reference'))
return scope.none if entry['operator'] == 'all'
values = values - [CountsAxes::MASKED_TEXT]
end
raise ArgumentError if [entry['value'], entry['end']].any? { !_1.nil? && _1 != '' }
parsed = values.map do |value|
@@ -200,7 +226,10 @@ class CountsCondition
operator = entry['operator']
if parsed.empty?
return ['in', 'any'].include?(operator) ? scope.none : scope
return scope.none if ['in', 'any'].include?(operator)
return scope if entry['values'].empty?
return @population.condition_scope(scope, field[:key])
end
source, expression = source_for(field[:key])
if field[:set_type] == 'tag_reference'
@@ -221,6 +250,7 @@ class CountsCondition
predicate = source ? "NOT #{ predicate }" :
"(NOT #{ predicate } OR #{ blank_sql(expression, field[:type]) })"
end
scope = @population.condition_scope(scope, field[:key])
scope.where(predicate, *parsed)
end
@@ -349,7 +379,7 @@ class CountsCondition
return @definition.alias_source(root) if root.start_with?('aliases_')
return [nil, @definition.name_expression(root)]
end
return [nil, "(#{ @population.latest_sql })"] if root == 'latest'
return [nil, "(#{ @population.current_latest_sql })"] if root == 'latest'
if root == 'user' && parts.any?
expression = parts.first == 'id' ? 'v.created_by_user_id' :
'(SELECT actor.name FROM users actor WHERE actor.id = v.created_by_user_id)'
+1 -1
ファイルの表示
@@ -20,7 +20,7 @@ class CountsDefaults
details = details.transform_values do |value|
value.split(' ').flat_map do |token|
if resource == 'tags' && ['name', 'parents.name'].include?(token)
token == 'name' ? name : ["parents.#{ name }", "parents.#{ aliases }"]
token == 'name' ? name : "parents.#{ name }"
elsif resource == 'tags' && token == 'aliases'
aliases
elsif resource == 'tags' && token.start_with?('aliases.')
+47
ファイルの表示
@@ -13,6 +13,53 @@ class CountsExportDownload
class TimedOut < StandardError; end
class StorageFull < StandardError; end
# Rack body deliberately has no to_path: sendfile middleware must not bypass ACK.
class ResponseBody
def initialize body, &state
@body = body
@closed = false
@state = state
end
def each &block
return enum_for(:each) unless block
return if @closed
transmit(&block)
end
def close
return if @closed
@closed = true
@body.close if @body.respond_to?(:close)
end
private
def transmit
first = true
@body.each do |chunk|
raise IOError, 'Download no longer authorised' unless @state.call('active')
yield chunk
if first
# Rack accepted a chunk; browser receipt and saving are not guaranteed.
raise IOError, 'Download acknowledgement failed' unless @state.call('started')
first = false
end
end
@state.call('failed') if first
rescue StandardError => error
Rails.logger.warn("counts_download_stream_failure: #{ error.class }")
@state.call('failed')
raise
ensure
close
end
end
def self.directory
Rails.root.join(ENV.fetch('COUNTS_EXPORT_DIRECTORY', 'tmp/counts_exports'))
end
+83 -3
ファイルの表示
@@ -45,6 +45,20 @@ class CountsExportJobs
write(state)
end
def self.failure_code(error)
case error
when Cancelled then 'cancelled'
when CountsAxes::TooManyAxes then 'too_many_axes'
when CountsAxes::PrivateCondition then 'private_condition'
when CountsTableQuery::TooManyColumns then 'too_many_columns'
when ActiveRecord::StatementTimeout then 'counts_query_timeout'
when CountsExportDownload::TimedOut then 'generation_timeout'
when CountsExportDownload::TooLarge then 'export_too_large'
when CountsExportDownload::StorageFull, Errno::ENOSPC then 'export_storage_full'
else 'generation_failed'
end
end
def self.directory = CountsExportDownload.directory.join('jobs')
def self.verifier = Rails.application.message_verifier('counts_export_job')
@@ -176,6 +190,12 @@ class CountsExportJobs
return unless state && state['owner'] == owner && state['expires_at'] > Time.current.to_f
expire_stalled(state)
if (state['status'] == 'ready' &&
['pending', 'streaming'].include?(state['download_status']) &&
state.fetch('download_expires_at', 0) <= Time.current.to_f)
state['download_status'] = 'timeout'
write(state)
end
if cancel && ['queued', 'running', 'ready'].include?(state['status'])
state['status'] = state['status'] == 'running' ? 'cancelling' : 'cancelled'
@@ -183,11 +203,71 @@ class CountsExportJobs
state.delete('token')
write(state)
end
state.slice('status', 'phase', 'processed', 'total', 'format', 'token', 'error',
'updated_at', 'heartbeat_at')
result = state.slice('status', 'phase', 'processed', 'total', 'format', 'token', 'error',
'updated_at', 'heartbeat_at', 'download_attempt', 'download_status')
result['max_axes'] = CountsAxes::MAX_AXES if state['error'] == 'too_many_axes'
result['max_columns'] = CountsTableQuery::MAX_COLUMNS if state['error'] == 'too_many_columns'
result
end
end
def self.download_attempt(token, owner, file_token: nil, attempt: nil, status: nil)
id = authorised_id(token, owner)
return unless id
synchronise do
state = read(id)
return unless (state && state['owner'] == owner && state['status'] == 'ready' &&
state['expires_at'] > Time.current.to_f && HistoryAccess.authorised_artifact?(state))
if attempt
return unless state['download_attempt'] == attempt && state['token'] == file_token
return unless state['download_status'] == 'started' ||
state.fetch('download_expires_at', 0) > Time.current.to_f
transitions = { 'streaming' => ['pending'], 'started' => ['streaming'],
'failed' => ['pending', 'streaming', 'started'],
'unavailable' => ['pending', 'streaming'] }
if status == 'active'
return state.slice('download_attempt', 'download_status', 'token') if
['streaming', 'started'].include?(state['download_status'])
return
end
if status
return unless transitions.fetch(status, []).include?(state['download_status'])
state['download_status'] = status
else
return unless state['download_status'] == 'pending'
end
else
# Repeated acceptance POSTs do not invalidate an in-flight GET or issue duplicates.
if (state['download_status'] == 'started' ||
(['pending', 'streaming'].include?(state['download_status']) &&
state.fetch('download_expires_at', 0) > Time.current.to_f))
return state.slice('download_attempt', 'download_status', 'token')
end
state['download_attempt'] = SecureRandom.hex(24)
state['download_status'] = 'pending'
state['download_expires_at'] = Time.current.to_f + 45
end
write(state)
state.slice('download_attempt', 'download_status', 'token')
end
end
# GET already verified authority. During streaming, observe cancellation without
# repeating the six-table policy revision aggregate for every 64KiB chunk.
def self.stream_active?(token, owner, file_token:, attempt:)
id = authorised_id(token, owner)
state = id && read(id)
state && state['owner'] == owner && state['status'] == 'ready' &&
state['expires_at'] > Time.current.to_f && state['token'] == file_token &&
state['download_attempt'] == attempt &&
(state['download_status'] == 'started' ||
(state['download_status'] == 'streaming' &&
state.fetch('download_expires_at', 0) > Time.current.to_f))
end
def self.checkpoint(id, processed = nil, total = nil, phase: nil)
synchronise do
state = read(id)
@@ -376,7 +456,7 @@ class CountsExportJobs
if current
unless current['status'] == 'failed'
current['status'] = error.is_a?(Cancelled) ? 'cancelled' : 'failed'
current['error'] = error.class.name
current['error'] = failure_code(error)
end
current['updated_at'] = Time.current.to_f
write(current)
+1 -1
ファイルの表示
@@ -157,7 +157,7 @@ class CountsExportProjection
def csv_display(value, key)
if key.start_with?('aliases_') && value.is_a?(Array)
return value.map { fetch(_1, 'name') }.compact.join("\n")
return value.map { fetch(_1, 'name') }.compact.join(' ')
end
return @labels.fetch('bot', '') if key == 'created_by_user' && value.nil?
return @labels.fetch("event_type:#{ value }", value) if key == 'event_type'
+8 -1
ファイルの表示
@@ -6,6 +6,13 @@ class CountsExporter
'summary_columns', 'nested_json', 'child_sort', 'labels'].freeze
attr_reader :format, :resource
def self.with_read_snapshot(&block)
# A caller-owned transaction already owns the read view (including fixtures).
# MySQL cannot change isolation inside it. Standalone workers always start RR.
options = ApplicationRecord.connection.transaction_open? ? {} : { isolation: :repeatable_read }
ApplicationRecord.transaction(**options, &block)
end
def initialize(params, language_code:)
params = CountsResourceDefinition.normalise_params(params)
if params.key?('definition')
@@ -52,7 +59,7 @@ class CountsExporter
return @table_export.body(progress:, phase:) if @table_export
Enumerator.new do |output|
ApplicationRecord.transaction(isolation: :repeatable_read) do
self.class.with_read_snapshot do
processed = 0
first_row = true
started = false
+1
ファイルの表示
@@ -28,6 +28,7 @@ class CountsPivotLayout
result = []
values.each_with_index do |tuple, index|
result << { values: tuple, kind: 'data' }
# A group continuing on the next page closes only on its final page.
following = values[index + 1] || next_values
depths.reverse_each do |depth|
prefix = tuple.first(depth)
+43 -18
ファイルの表示
@@ -1,5 +1,6 @@
# Re-aggregate the population, never add up already grouped cells (multi-value axes overlap).
# Re-aggregate distinct membership; tag row tuples alone use additive contributions.
class CountsPivotTotals
class TimedOut < ActiveRecord::StatementTimeout; end
def initialize(population, axes, row_headers, column_headers)
@population = population
@axes = axes
@@ -8,8 +9,12 @@ class CountsPivotTotals
end
def call
# MySQL's unlimited LIMIT prevents CTE merging: authorise once for all branches.
basis = @population.scope.except(:select, :order).select('v.*')
# Materialise full-cell membership once, including omitted multi-value axes.
membership = @population.scope.except(:select, :order)
joins = @axes.joins
membership = membership.joins(joins) unless joins.empty?
# Retain exactly the records contributing a full cell, including changed_field.
basis = membership.select('v.*').distinct
.limit(18_446_744_073_709_551_615).to_sql
queries = @rows.flat_map do |row_depth, headers|
@columns.filter_map do |column_depth, columns|
@@ -18,41 +23,59 @@ class CountsPivotTotals
indexes = (0...row_depth).to_a +
(0...column_depth).map { @axes.rows.length + _1 }
expressions = indexes.map { @axes.expressions[_1] }
scope = @population.scope.except(:select, :order, :where, :from).from('authorised v')
joins = @axes.joins(indexes.map { @axes.all[_1].fetch('field') })
scope = @population.scope.except(:select, :order, :where, :from).from('population_members v')
additive = @axes.rows.any? { _1['field'] == 'tags' }
retained = additive ? (0...@axes.rows.length).to_a | indexes : indexes
joins = @axes.joins(retained.map { @axes.all[_1].fetch('field') })
scope = scope.joins(joins) unless joins.empty?
predicates = []
if row_depth.positive?
tuples = headers.map { _1[:values] }.uniq
predicate = tuples.map { Array.new(row_depth) { |i|
"(#{ expressions[i] }) <=> ?"
"a#{ indexes[i] } <=> ?"
}.join(' AND ') }.map { "(#{ _1 })" }.join(' OR ')
scope = scope.where(predicate, *tuples.flatten)
predicates << @population.definition.model.sanitize_sql_array([
tuples.empty? ? 'FALSE' : "(#{ predicate })", *tuples.flatten])
end
if column_depth.positive?
tuples = columns.map { _1[:values] }.uniq
predicate = tuples.map { Array.new(column_depth) { |i|
"(#{ expressions[row_depth + i] }) <=> ?"
"a#{ indexes[row_depth + i] } <=> ?"
}.join(' AND ') }.map { "(#{ _1 })" }.join(' OR ')
scope = scope.where(predicate, *tuples.flatten)
predicates << @population.definition.model.sanitize_sql_array([
tuples.empty? ? 'FALSE' : "(#{ predicate })", *tuples.flatten])
end
grouped = expressions.each_with_index.map { |expression, index|
"#{ expression } AS v#{ index }"
# Filter projected coordinates, not repeated masking/time expressions.
# Each member still participates in the complete DISTINCT population.
selection = predicates.empty? ? '' : " WHERE #{ predicates.join(' AND ') }"
grouped = indexes.each_with_index.map { |key, index|
"a#{ key } AS v#{ index }"
}
# Fixed-width result coordinates, independent of NULL and of translated labels.
padding = (expressions.length...@axes.all.length).map { "NULL AS v#{ _1 }" }
fields = ["#{ row_depth } AS row_depth", "#{ column_depth } AS column_depth"] +
grouped + padding + ['COUNT(DISTINCT v.id) AS count']
scope = scope.group(*expressions.map { Arel.sql(_1) }) if expressions.any?
scope.optimizer_hints('MAX_EXECUTION_TIME(10000)')
.select(Arel.sql(fields.join(', '))).to_sql
grouped + padding + ['count']
if additive
# Sum full row tuples, not page cells; preserve DISTINCT inside each tuple.
inner = @axes.grouped_sql(scope, retained)
coordinates = indexes.map { "a#{ _1 }" }
projected = coordinates.each_with_index.map { |key, index| "#{ key } AS v#{ index }" }
output = ["#{ row_depth } AS row_depth", "#{ column_depth } AS column_depth"] +
projected + padding + ['SUM(count) AS count']
grouping = coordinates.empty? ? '' : " GROUP BY #{ coordinates.join(', ') }"
"SELECT #{ output.join(', ') } FROM (#{ inner }) leaf#{ selection }#{ grouping }"
else
inner = @axes.grouped_sql(scope, indexes)
"SELECT #{ fields.join(', ') } FROM (#{ inner }) grouped#{ selection }"
end
end
end
return [] if queries.empty?
sql = queries.map { "(#{ _1 })" }.join(' UNION ALL ')
@population.definition.model.connection.select_all(
"WITH authorised AS (#{ basis }) " +
"SELECT /*+ MAX_EXECUTION_TIME(10000) */ * FROM (#{ sql }) totals").map do |record|
statement = "WITH population_members AS (#{ basis }) " +
"SELECT /*+ #{ CountsTableQuery::SQL_HINTS.join(' ') } */ * FROM (#{ sql }) totals"
@population.definition.model.connection.select_all(statement).map do |record|
row_depth = record['row_depth'].to_i
column_depth = record['column_depth'].to_i
{ row: (0...row_depth).map { record["v#{ _1 }"] },
@@ -62,5 +85,7 @@ class CountsPivotTotals
row_depth, column_depth),
count: record['count'].to_i }
end
rescue ActiveRecord::StatementTimeout
raise TimedOut, 'Counts pivot totals exceeded the query time limit'
end
end
+90 -41
ファイルの表示
@@ -41,8 +41,7 @@ class CountsPopulation
raise ArgumentError
end
def latest_sql(row = 'v', cutoff = nil)
cutoff ||= history ? at : timestamp(params['at']) || at
def latest_at_sql(row, cutoff)
keys = definition.keys.map { "newer.#{ _1 } = #{ row }.#{ _1 }" }.join(' AND ')
bound = definition.model.connection.quote(cutoff)
<<~SQL.squish
@@ -52,16 +51,73 @@ class CountsPopulation
SQL
end
def current_latest_sql(row = 'v')
keys = definition.keys.map { "newer.#{ _1 } = #{ row }.#{ _1 }" }.join(' AND ')
<<~SQL.squish
NOT EXISTS (SELECT 1 FROM #{ definition.table } newer
WHERE #{ keys } AND newer.version_no > #{ row }.version_no)
SQL
end
def changed_sql(key)
column = definition.column(key)
"(previous.id IS NOT NULL AND NOT (v.#{ column } <=> previous.#{ column }))"
end
# Unrestricted predicates are used only for numerical totals, never record output.
# Aggregate sources are unrestricted; CountsAxes masks their output coordinates.
# Never serialise raw records from an unrestricted population.
def access_sql(model, row)
@authorisation_flg ? HistoryAccess.sql(model, row) : 'TRUE'
end
# Each predicate keeps its own visibility restriction, including OR branches.
def condition_scope(scope, field)
return scope if HistoryAccess.admin? || !CountsAxes.private_field?(field)
scope.where(Arel.sql(CountsAxes.visibility_sql(definition, field)))
end
def axis_condition(scope, entry)
raise CountsAxes::PrivateCondition if entry['value'] == CountsAxes::MASKED_VALUE
key = entry.fetch('field')
raise ArgumentError unless definition.axes(history).include?(key)
value = entry['value']
unless value.nil?
valid = case definition.fields[key]
when :integer
value.is_a?(Integer) || (value.is_a?(String) && value.match?(/\A-?[0-9]+\z/))
when :boolean then [true, false].include?(value)
else value.is_a?(String)
end
raise ArgumentError unless valid
end
if entry.key?('grain')
unless definition.fields[key] == :datetime && CountsAxes::GRAINS.include?(entry['grain'])
raise ArgumentError
end
end
if key == 'tags'
scope = scope.where(snapshot_membership(entry.fetch('value')))
elsif key == 'changed_field'
raise ArgumentError unless history && definition.changes.include?(entry['value'])
scope = scope.where(Arel.sql(changed_sql(entry['value'])))
elsif definition.fields[key] == :datetime && entry['grain']
scope = scope.where(definition.expression(key) => nil) if entry['value'].nil?
if entry['value']
expression = CountsAxes.time_expression(definition.expression(key),
entry['grain'], timezone)
scope = scope.where("#{ expression } = ?", entry['value'])
end
else
raise ArgumentError unless definition.axes(history).include?(key)
scope = scope.where("#{ definition.expression(key) } <=> ?", entry['value'])
end
scope
end
private
def make_population
@@ -69,7 +125,8 @@ class CountsPopulation
@scope = @scope.from("#{ table } v").select('v.*')
previous = ['rows', 'cols'].any? do |key|
QueryState.axes(params.fetch(key, '')).any? { _1['field'] == 'changed_field' }
end || QueryState.records(params, 'drill').any? { _1['field'] == 'changed_field' }
end || (QueryState.records(params, 'drill') + QueryState.records(params, 'conditions'))
.any? { _1['field'] == 'changed_field' }
if previous
keys = definition.keys.map { "previous.#{ _1 } = v.#{ _1 }" }.join(' AND ')
@scope = @scope.joins("LEFT JOIN #{ table } previous ON #{ keys } " +
@@ -82,7 +139,7 @@ class CountsPopulation
raise ArgumentError if point > at
@scope = @scope.where("v.#{ definition.time_column } <= ?", point)
.where(Arel.sql(latest_sql('v', point))).where.not('v.event_type' => 'discard')
.where(Arel.sql(latest_at_sql('v', point))).where.not('v.event_type' => 'discard')
return
end
first = timestamp(params['from'])
@@ -106,6 +163,14 @@ class CountsPopulation
end
apply_tags if params['tags'].present? || params['tag'].present?
conditions = QueryState.records(params, 'conditions')
drills = QueryState.records(params, 'drill')
raise ArgumentError unless drills.length <= 100
drills.each do |entry|
raise ArgumentError unless (entry.keys - ['field', 'grain', 'value']).empty?
condition = entry.merge('operator' => 'axis_eq')
conditions << condition unless conditions.include?(condition)
end
raise ArgumentError unless conditions.is_a?(Array) && conditions.length <= 100
values = QueryState.records(params, 'condition_values')
@@ -114,7 +179,13 @@ class CountsPopulation
raise ArgumentError unless index.between?(0, conditions.length - 1)
raise ArgumentError unless conditions[index]['values'] == true
end
evaluated = []
conditions.each_with_index do |entry, index|
if entry['operator'] == 'axis_eq'
next if evaluated.include?(entry)
evaluated << entry
end
if entry.key?('values')
raise ArgumentError unless entry['values'] == true
@@ -123,30 +194,7 @@ class CountsPopulation
end
apply_condition(entry)
end
drills = QueryState.records(params, 'drill')
raise ArgumentError unless drills.is_a?(Array) && drills.length <= 100
drills.each do |entry|
key = entry.fetch('field')
if key == 'tags'
@scope = @scope.where(snapshot_membership(entry.fetch('value')))
elsif key == 'changed_field'
raise ArgumentError unless history && definition.changes.include?(entry['value'])
@scope = @scope.where(Arel.sql(changed_sql(entry['value'])))
elsif definition.fields[key] == :datetime && entry['grain']
@scope = @scope.where(definition.expression(key) => nil) if entry['value'].nil?
if entry['value']
expression = CountsAxes.time_expression(definition.expression(key),
entry['grain'], timezone)
@scope = @scope.where("#{ expression } = ?", entry['value'])
end
else
raise ArgumentError unless definition.axes(history).include?(key)
@scope = @scope.where(definition.expression(key) => entry['value'])
end
end
end
def apply_condition(entry)
@@ -168,27 +216,28 @@ class CountsPopulation
def apply_tags
raise ArgumentError unless definition.resource == 'posts'
@scope = condition_scope(@scope, 'tags')
if params['tag'].present?
@scope = @scope.where(snapshot_membership(
"#{ params.fetch('kind', 'internal') }:#{ params['tag'] }"))
end
return unless params['tags'].present?
refs = PostTagSearch.parse_references(params, params['tags'].split)
raw = params['tags']
literals = PostTagSearch.literals(nil, raw, references: refs)
tokens = raw.split.reject { _1.downcase == 'type:or' }
locale = Locale.find_by(code: params.fetch('locale', 'ja'))
raise ArgumentError unless locale
literals = PostTagSearch.literals(locale, tokens.join(' '))
identities = PostTagSearch.references_for(locale, literals)
predicates = literals.map do |literal|
reference = literal[:reference]
expression = if reference
parts = { 'internal' => reference['tag_id'],
'external' => reference['external_tag_id'] }.filter_map do |kind, id|
snapshot_membership("#{ kind }:#{ id }") if id
end
parts.empty? ? 'FALSE' : "(#{ parts.join(' OR ') })"
else
@condition ||= CountsCondition.new(self)
@condition.tag_name_match(literal[:name])
end
# Resolve current names/aliases exactly as normal post search does. Only
# membership comes from history; snapshot names never identify a tag.
internal, external = identities.fetch(literal[:name])
parts = { 'internal' => internal, 'external' => external }.flat_map do |kind, ids|
Array(ids).map { |id| snapshot_membership("#{ kind }:#{ id }") }
end
expression = parts.empty? ? 'FALSE' : "(#{ parts.join(' OR ') })"
literal[:negative] ? "NOT #{ expression }" : expression
end
return if predicates.empty?
+18 -3
ファイルの表示
@@ -105,12 +105,25 @@ class CountsResourceDefinition
def time_column = 'created_at'
def column(key)
raise ArgumentError unless fields.key?(key)
# Keep the V1 Counts field name at the API boundary only.
return 'linked_tag_ids' if resource == 'nico_tags' && key == 'linked_tags'
{ 'resource_id' => keys.first, 'user' => 'created_by_user_id' }.fetch(key, key)
end
def expression(key, row = 'v') = "#{ row }.#{ column(key) }"
def changes = fields.keys - COMMON.keys - ['updated_at', 'updated_by_user_id']
def expression(key, row = 'v')
if resource == 'nico_tags' && key == 'name'
return "(SELECT CONCAT(external_tags.platform, ':', external_tags.name) " +
"FROM external_tags WHERE external_tags.id = #{ row }.tag_id)"
end
"#{ row }.#{ column(key) }"
end
def changes
columns = fields.keys - COMMON.keys - ['updated_at', 'updated_by_user_id']
resource == 'nico_tags' ? columns - ['name'] : columns
end
def axes(history)
fields.reject { |_key, type| type == :json }.keys +
(resource == 'posts' ? ['tags'] : []) + (history ? ['changed_field'] : [])
@@ -118,7 +131,9 @@ class CountsResourceDefinition
def metadata
fields.map do |key, type|
{ key:, type:, nullable: model.columns_hash.fetch(column(key)).null,
nullable = (resource == 'nico_tags' && key == 'name') ||
model.columns_hash.fetch(column(key)).null
{ key:, type:, nullable:,
operators: OPERATORS.fetch(type, []) }
end
end
+3 -1
ファイルの表示
@@ -201,6 +201,8 @@ class CountsStateQuery
end
def listing
external_name = CountsResourceDefinition.new('nico_tags')
.expression('name', 'external_version')
catalog = <<~SQL
, tag_counts AS (
SELECT membership.kind, membership.tag_id, COUNT(*) AS post_count
@@ -209,7 +211,7 @@ class CountsStateQuery
SELECT 'internal' AS kind, tag_id, name, names,
deprecated_at FROM tag_at WHERE event_type <> 'discard'
UNION ALL
SELECT tc.kind, tc.tag_id, external_version.name, JSON_OBJECT() AS names,
SELECT tc.kind, tc.tag_id, #{ external_name } AS name, JSON_OBJECT() AS names,
NULL AS deprecated_at
FROM tag_counts tc
LEFT JOIN tag_at internal_version
+3 -1
ファイルの表示
@@ -57,7 +57,7 @@ class CountsTableExport
def body(progress: nil, phase: nil)
Enumerator.new do |output|
ApplicationRecord.transaction(isolation: :repeatable_read) do
CountsExporter.with_read_snapshot do
processed = 0
started = false
first = true
@@ -253,6 +253,8 @@ class CountsTableExport
end
def summary_value(value, index, page)
return CountsAxes::MASKED_TEXT if value == CountsAxes::MASKED_VALUE
field = @query.axes.all.fetch(index).fetch('field')
return @labels.fetch('bot', '') if field == 'user' && value.nil?
if ['language_code', 'source_language_code'].include?(field)
+114 -42
ファイルの表示
@@ -1,10 +1,13 @@
# Common entrypoint for six version resources; legacy endpoints keep their contracts.
class CountsTableQuery
SQL_TIMEOUT_SECONDS = 30
SQL_HINTS = ["MAX_EXECUTION_TIME(#{ SQL_TIMEOUT_SECONDS * 1000 })",
"SET_VAR(optimizer_switch='semijoin=off')"].freeze
MAX_COLUMNS = 100
class TooManyColumns < ArgumentError; end
KEYS = ['resource', 'definition', 'mode', 'at', 'execution_at', 'from', 'through',
'timezone', 'rows', 'cols', 'conditions', 'drill', 'tags', 'tag_refs', 'match',
'timezone', 'rows', 'cols', 'conditions', 'drill', 'tags', 'match',
'tag', 'kind', 'resource_id', 'user_id', 'page', 'limit', 'sort', 'locale',
'measures', 'column_order', 'condition_values', 'name_scope'].freeze
attr_reader :population, :axes, :page, :limit, :mode
@@ -16,7 +19,9 @@ class CountsTableQuery
params = CountsDefaults.for(params).slice(*KEYS).merge(params)
@params = params
@population = CountsPopulation.new(params)
@mode = params.fetch('mode', 'summary')
validate_summary_conditions!
@population = CountsPopulation.new(params, authorisation_flg: mode != 'summary')
@definition = population.definition
@connection = @definition.model.connection
@axes = CountsAxes.new(population)
@@ -47,23 +52,30 @@ class CountsTableQuery
result = metadata.merge(mode == 'summary' ? summary : details)
visible = mode == 'summary' ? result[:visible_output_count] : result[:count]
totals = output_totals(visible)
result.merge(visible_output_count: visible,
response = result.merge(visible_output_count: visible,
**totals,
output_count_scope: 'before_authorisation',
output_count_unit: 'regular_result_rows',
page_grand_total_repeated: mode == 'summary' && axes.rows.any? &&
result[:group_count] > limit)
if mode == 'summary'
response.except!(:excluded_count, :excluded_count_scope)
response[:column_values_editable] = HistoryAccess.admin? ||
!result.fetch(:column_values, []).flatten.include?(CountsAxes::MASKED_VALUE)
end
response
end
end
def metadata
{ population_count: nil, population_count_computed: false,
{ condition_labels: condition_labels, population_count: nil, population_count_computed: false,
excluded_count: nil, excluded_count_scope: 'current_query',
population_count_scope: 'all_recorded_versions',
page:, limit:, execution_at: population.at.iso8601(6), at: @params['at'].presence,
definition: population.history ? 'history' : 'latest',
timezone: population.timezone.identifier, rows: axes.rows, cols: axes.columns,
fields: @definition.metadata,
masked_value: HistoryAccess.admin? ? nil : CountsAxes::MASKED_VALUE,
count: record_counts[:visible] }
end
@@ -75,19 +87,9 @@ class CountsTableQuery
cells: page == 1 ? [{ values: [], count: record_counts[:visible] }] : [],
axis_labels: {} }
end
expressions = axes.expressions
selected = expressions.each_with_index.map do |expression, index|
"#{ expression } AS a#{ index }"
end
source = population.scope
source = summary_population.scope
source = source.joins(axes.joins) unless axes.joins.empty?
sql = if expressions.empty?
source.except(:select).select('COUNT(DISTINCT v.id) AS count').to_sql
else
source.except(:select).group(*expressions.map { Arel.sql(_1) })
.select(Arel.sql((selected + ['COUNT(DISTINCT v.id) AS count']).join(', ')))
.to_sql
end
sql = axes.grouped_sql(source)
prefix = "WITH cells AS (#{ sql })"
column_values = limited_columns(prefix)
row_keys = axes.rows.each_index.map { "a#{ _1 }" }
@@ -128,7 +130,7 @@ class CountsTableQuery
numbered AS (SELECT row_groups.*,
ROW_NUMBER() OVER (ORDER BY #{ order_rows.join(', ') }) AS position FROM row_groups),
group_totals AS (SELECT COUNT(*) AS group_count FROM row_groups)
SELECT /*+ MAX_EXECUTION_TIME(10000) */ cells.*,
SELECT /*+ #{ SQL_HINTS.join(' ') } */ cells.*,
page_rows.position AS counts_position, group_totals.group_count AS counts_group_count
FROM group_totals LEFT JOIN numbered page_rows
ON page_rows.position > #{ (page - 1) * limit }
@@ -165,7 +167,7 @@ class CountsTableQuery
other_count: axes.rows.length, direction: :column)
.map { _1.merge(kind: 'grand_total') }
end
totals = CountsPivotTotals.new(population, axes, row_headers, column_headers).call
totals = CountsPivotTotals.new(summary_population, axes, row_headers, column_headers).call
{ group_count: total,
visible_output_count: axes.rows.empty? ? 1 : total,
column_values:, row_headers:, column_headers:, pivot_totals: totals,
@@ -175,11 +177,12 @@ class CountsTableQuery
end
def details
query = population.scope.select(Arel.sql(
"(#{ population.latest_sql }) AS counts_latest"))
query = population.scope.optimizer_hints(*SQL_HINTS).select(Arel.sql(
"(#{ population.current_latest_sql }) AS counts_latest"))
order = sort_sql { detail_expression(_1) }
query = query.order(Arel.sql((order + ['v.id ASC']).join(', ')))
.limit(limit).offset((page - 1) * limit)
query = query.preload(:external_tag) if @definition.resource == 'nico_tags'
rows = query.to_a
if ['posts', 'tags'].include?(@definition.resource)
point = population.history ? population.at :
@@ -218,21 +221,39 @@ class CountsTableQuery
private
# Private predicates require visibility independently of their operator.
# Ordering/measure state must not select a server-only masked coordinate.
def validate_summary_conditions!
return if mode != 'summary' || HistoryAccess.admin?
['column_order', 'measures'].each do |key|
# Coordinates are compared only to already masked expressions, never raw
# values. Public coordinates remain sortable; a guessed private ID cannot match.
raise CountsAxes::PrivateCondition if QueryState.records(@params, key).any? do |record|
record['value'] == CountsAxes::MASKED_VALUE
end
end
end
def unrestricted_population
@unrestricted_population ||= CountsPopulation.new(
@params.merge('execution_at' => population.at.iso8601(6)), authorisation_flg: false)
end
def summary_population
population
end
def record_counts
return @record_counts if @record_counts
if HistoryAccess.admin?
if HistoryAccess.admin? || mode == 'summary'
total = population.scope.except(:select)
.optimizer_hints('MAX_EXECUTION_TIME(10000)').count
.optimizer_hints(*SQL_HINTS).count
return @record_counts = { total:, visible: total, denied: 0 }
end
total, denied = unrestricted_population.scope.except(:select)
.optimizer_hints('MAX_EXECUTION_TIME(10000)').pick(Arel.sql('COUNT(*)'),
.optimizer_hints(*SQL_HINTS).pick(Arel.sql('COUNT(*)'),
Arel.sql("COALESCE(SUM(CASE WHEN #{ HistoryAccess.sql(@definition.model, 'v') } " +
'THEN 0 ELSE 1 END), 0)'))
# Related/previous-version predicates may differ under public authority.
@@ -242,14 +263,17 @@ class CountsTableQuery
total.to_i - denied.to_i
else
population.scope.except(:select)
.optimizer_hints('MAX_EXECUTION_TIME(10000)').count
.optimizer_hints(*SQL_HINTS).count
end
@record_counts = { total: total.to_i, denied: denied.to_i, visible: }
end
def output_totals(visible_rows)
def output_totals visible_rows
return @output_totals if @output_totals
if mode == 'summary'
return @output_totals = { output_count: visible_rows, excluded_count: nil }
end
if HistoryAccess.admin?
return @output_totals = { output_count: visible_rows, excluded_count: 0 }
end
@@ -281,7 +305,7 @@ class CountsTableQuery
prefix = "WITH cells AS (#{ sql })"
@connection.select_value(<<~SQL).to_i
#{ prefix }
SELECT /*+ MAX_EXECUTION_TIME(10000) */ COUNT(*) FROM (
SELECT /*+ #{ SQL_HINTS.join(' ') } */ COUNT(*) FROM (
SELECT #{ keys.join(', ') } FROM cells GROUP BY #{ keys.join(', ') }
) output_groups
SQL
@@ -289,7 +313,7 @@ class CountsTableQuery
def locale = @locale ||= Locale.find_by!(code: @params.fetch('locale', 'ja'))
def limited_columns(prefix)
def limited_columns prefix
keys = axes.columns.each_index.map { "a#{ axes.rows.length + _1 }" }
return [[]] if keys.empty?
@@ -307,7 +331,7 @@ class CountsTableQuery
end
values = @connection.select_rows(<<~SQL)
#{ prefix }
SELECT /*+ MAX_EXECUTION_TIME(10000) */ DISTINCT #{ keys.join(', ') } FROM cells
SELECT /*+ #{ SQL_HINTS.join(' ') } */ DISTINCT #{ keys.join(', ') } FROM cells
ORDER BY #{ order } LIMIT #{ MAX_COLUMNS + 1 }
SQL
raise TooManyColumns if values.length > MAX_COLUMNS
@@ -315,9 +339,13 @@ class CountsTableQuery
values
end
def measure_relation(values)
row_expressions = axes.expressions.first(axes.rows.length)
source = population.scope.except(:select, :order)
def measure_relation values
source = summary_population.scope.except(:select, :order)
# Omitted multi-value column axes still determine full-cell membership.
membership = summary_population.scope.except(:select, :order)
all_joins = axes.joins
membership = membership.joins(all_joins) unless all_joins.empty?
source = source.where("v.id IN (#{ membership.select('v.id').distinct.to_sql })")
fields = axes.rows + axes.columns.first(values.length)
joins = axes.joins(fields.map { _1.fetch('field') })
source = source.joins(joins) unless joins.empty?
@@ -325,17 +353,23 @@ class CountsTableQuery
expression = axes.expressions[axes.rows.length + index]
source = source.where("(#{ expression }) <=> ?", value)
end
selected = row_expressions.each_with_index.map { |expression, index|
"#{ expression } AS a#{ index }"
} + ['COUNT(DISTINCT v.id) AS count']
source = source.group(*row_expressions.map { Arel.sql(_1) }) if row_expressions.any?
source.select(Arel.sql(selected.join(', '))).to_sql
axes.grouped_sql(source, axes.rows.each_index.to_a)
end
def axis_order_sql(key, direction = 'asc', qualifier: nil)
def axis_order_sql key, direction = 'asc', qualifier: nil
column = qualifier ? "#{ qualifier }.#{ key }" : key
expression = column
if axes.all.fetch(key.delete_prefix('a').to_i).fetch('field') == 'category'
field = axes.all.fetch(key.delete_prefix('a').to_i).fetch('field')
if !HistoryAccess.admin? && [:integer, :boolean].include?(@definition.fields[field])
masked = @connection.quote(CountsAxes::MASKED_VALUE)
expression = "CASE WHEN #{ column } <=> #{ masked } THEN NULL " +
"ELSE CAST(#{ column } AS SIGNED) END"
return "#{ column } IS NULL ASC, (#{ column } <=> #{ masked }) ASC, " +
"#{ expression } #{ direction.upcase }"
end
if field == 'event_type'
expression = event_order_sql(column)
elsif field == 'category'
# Tag's enum preserves the same category order as TagsController.
cases = Tag.categories.keys.each_with_index.map { |category, index|
"WHEN #{ @connection.quote(category) } THEN #{ index }"
@@ -345,7 +379,7 @@ class CountsTableQuery
"#{ column } IS NULL ASC, #{ expression } #{ direction.upcase }"
end
def measure_values(key)
def measure_values key
records = QueryState.records(@params, 'measures').select { _1['key'] == key }
raise ArgumentError if records.empty?
raise ArgumentError unless records.map { _1['axis'] }.sort == (0...records.length).to_a
@@ -359,7 +393,8 @@ class CountsTableQuery
end
def detail_expression(key)
return "(#{ population.latest_sql })" if key == 'latest'
return "(#{ population.current_latest_sql })" if key == 'latest'
return event_order_sql(@definition.expression(key)) if key == 'event_type'
return @definition.name_expression(key) if @definition.resource == 'tags' &&
@definition.name_column(key)
aliases = { 'tags' => 'tags_json', 'parents' => @definition.resource == 'posts' ?
@@ -371,6 +406,11 @@ class CountsTableQuery
"CAST(#{ expression } AS CHAR) COLLATE utf8mb4_bin"
end
def event_order_sql(expression)
"CASE #{ expression } WHEN 'create' THEN 0 WHEN 'update' THEN 1 " +
"WHEN 'discard' THEN 2 WHEN 'restore' THEN 3 ELSE 4 END"
end
def sort_sql
@sort.flat_map do |key, direction|
expression = yield key
@@ -378,6 +418,25 @@ class CountsTableQuery
end
end
def condition_labels
entries = QueryState.records(@params, 'conditions').select { _1['operator'] == 'axis_eq' }
entries += QueryState.records(@params, 'drill')
entries.group_by { _1['field'] }.to_h do |field, conditions|
values = conditions.map { _1['value'] }.compact.uniq
labels = case field
when 'tags' then tag_labels(values)
when 'parent_tag_ids' then parent_tag_labels(values)
when 'user'
actors = HistoryAccess.visible(population.scope, 'v').except(:select, :order)
.select('v.created_by_user_id')
User.where(id: values).where(id: actors)
.pluck(:id, :name).to_h.transform_keys(&:to_s)
else {}
end
[field, labels]
end
end
def axis_labels(records, column_values)
result = {}
@axis_names = {}
@@ -386,7 +445,7 @@ class CountsTableQuery
if index >= axes.rows.length
values += column_values.map { _1[index - axes.rows.length] }
end
values = values.compact.uniq
values = values.compact.uniq - [CountsAxes::MASKED_VALUE]
labels = case axis['field']
when 'user'
User.where(id: values).pluck(:id, :name).to_h.transform_keys(&:to_s)
@@ -396,6 +455,7 @@ class CountsTableQuery
labels = tag_labels(values)
@axis_names["a#{ index }"] = @tag_names
labels
when 'parent_tag_ids' then parent_tag_labels(values)
else {}
end
result["a#{ index }"] = labels
@@ -403,6 +463,17 @@ class CountsTableQuery
result
end
def parent_tag_labels(values)
ids = values.flat_map { _1.to_s.split }.uniq
tag_labels(ids.map { "internal:#{ _1 }" })
values.to_h do |value|
names = value.to_s.split.map do |id|
@tag_names.fetch("internal:#{ id }", {}).dig(locale.language_code, 'name')
end
[value.to_s, names.compact.join(' ')]
end
end
def tag_labels(values)
internal = values.grep(/\Ainternal:/).map { _1.split(':').last.to_i }
external = values.grep(/\Aexternal:/).map { _1.split(':').last.to_i }
@@ -416,10 +487,11 @@ class CountsTableQuery
table = model.table_name
scope = HistoryAccess.visible(model.where(tag_id: ids)).where('created_at <= ?', point)
scope = scope.includes(:external_tag) if model == NicoTagVersion
bound = @connection.quote(point)
if population.history && @definition.resource == 'posts'
key = kind == 'internal' ? 'tag_id' : 'external_tag_id'
pairs = population.scope.except(:select, :order).joins(
pairs = HistoryAccess.visible(population.scope, 'v').except(:select, :order).joins(
"JOIN JSON_TABLE(v.tags_json, '$[*]' COLUMNS (tag_id BIGINT PATH '$.#{ key }', " +
"version_no INT PATH '$.version_no' NULL ON EMPTY)) recorded_tag")
.where('recorded_tag.tag_id IN (?)', ids).group('recorded_tag.tag_id')
+2 -1
ファイルの表示
@@ -51,7 +51,8 @@ class MaterialSyncImporter
suppression: @import_block)
end
Material.transaction do
VersionRecorder.record_operation!(
event_type: "materials.#{ event_type }", created_by_user: @attributes[:updated_by_user]) do
if material.persisted?
MaterialVersionRecorder.ensure_snapshot!(material,
created_by_user: @attributes[:updated_by_user])
+17 -13
ファイルの表示
@@ -3,14 +3,16 @@ class MaterialSyncRunner
:errors, keyword_init: true)
def self.sync_enabled!
results = MaterialSyncSource.enabled.order(:id).map { |source| new(source).sync! }
VersionRecorder.with_event(event_type: 'materials.sync', created_by_user: nil) do
results = MaterialSyncSource.enabled.order(:id).map { |source| new(source).sync! }
Result.new(imported: results.sum(&:imported),
updated: results.sum(&:updated),
unchanged: results.sum(&:unchanged),
suppressed: results.sum(&:suppressed),
failed: results.sum(&:failed),
errors: results.flat_map(&:errors))
Result.new(imported: results.sum(&:imported),
updated: results.sum(&:updated),
unchanged: results.sum(&:unchanged),
suppressed: results.sum(&:suppressed),
failed: results.sum(&:failed),
errors: results.flat_map(&:errors))
end
end
def initialize source
@@ -21,13 +23,15 @@ class MaterialSyncRunner
result = Result.new(imported: 0, updated: 0, unchanged: 0,
suppressed: 0, failed: 0, errors: [])
if @source.source_kind == 'google_drive_path'
sync_google_drive_path!(result)
else
candidates.each do |candidate|
next if candidate.blank?
VersionRecorder.with_event(event_type: 'materials.sync', created_by_user: nil) do
if @source.source_kind == 'google_drive_path'
sync_google_drive_path!(result)
else
candidates.each do |candidate|
next if candidate.blank?
sync_candidate!(candidate, result)
sync_candidate!(candidate, result)
end
end
end
+2 -1
ファイルの表示
@@ -11,7 +11,8 @@ class MaterialSyncSuppressionRegistrar
def create!
suppression = nil
MaterialSyncSuppression.transaction do
VersionRecorder.record_operation!(
event_type: 'materials.block', created_by_user: @created_by_user) do
suppression = MaterialSyncSuppression.create!(
@attributes.merge(created_by_user: @created_by_user))
discard_existing_materials!(suppression)
+9 -5
ファイルの表示
@@ -1,20 +1,24 @@
class MaterialVersionRecorder < VersionRecorder
EVENT_TYPES = ['create', 'update', 'discard', 'restore'].freeze
def self.record! material:, event_type:, created_by_user:, file_snapshot: nil
new(material:, event_type:, created_by_user:, file_snapshot:).record!
def self.record! material:, event_type:, created_by_user:, event: nil, file_snapshot: nil
new(material:, event_type:, event:, created_by_user:, file_snapshot:).record!
end
def initialize material:, event_type:, created_by_user:, file_snapshot: nil
def initialize material:, event_type:, created_by_user:, event: nil, file_snapshot: nil
@file_snapshot = file_snapshot
super(record: material, event_type:, created_by_user:)
super(record: material, event_type:, event:, created_by_user:)
end
def self.ensure_snapshot! material, created_by_user:
return if material.material_versions.exists?
record!(material:, event_type: :create,
event = current_event || Event.new(
event_type: 'materials.create',
created_by_user: material.created_by_user || created_by_user,
created_at: material.created_at)
record!(material:, event_type: :create, event:,
created_by_user: material.created_by_user || created_by_user)
end
+6 -6
ファイルの表示
@@ -1,10 +1,11 @@
class NicoTagVersionRecorder < VersionRecorder
def self.record! external_tag:, event_type:, created_by_user:
new(external_tag:, event_type:, created_by_user:).record!
EVENT_TYPES = ['create', 'update'].freeze
def self.record! external_tag:, event_type:, created_by_user:, event: nil
new(external_tag:, event_type:, created_by_user:, event:).record!
end
def initialize external_tag:, event_type:, created_by_user:
super(record: external_tag, event_type:, created_by_user:)
def initialize external_tag:, event_type:, created_by_user:, event: nil
super(record: external_tag, event_type:, created_by_user:, event:)
end
private
@@ -14,8 +15,7 @@ class NicoTagVersionRecorder < VersionRecorder
def record_key = :external_tag
def snapshot_attributes
{ name: "#{ @record.platform }:#{ @record.name }",
linked_tags: @record.snapshot_linked_tag_names.join(' ') }
{ linked_tag_ids: @record.snapshot_linked_tag_ids.join(' ') }
end
def tracks_version_no_on_record? = false
+14 -5
ファイルの表示
@@ -9,6 +9,8 @@ class PostBulkCreator
def run locale
results = Array.new(@posts.length)
mutex = Mutex.new
event_mutex = Mutex.new
event = Event.new(event_type: 'posts.create', created_by_user_id: @actor_id)
next_index = 0
workers = Array.new(2) do
@@ -27,7 +29,7 @@ class PostBulkCreator
break if index >= @posts.length
attributes = @posts[index]
results[index] = create_row(locale, actor, attributes, index)
results[index] = create_row(locale, actor, attributes, index, event, event_mutex)
rescue StandardError => e
Rails.logger.error(
"post_bulk_creator_worker_failure #{ { error: e.class.name,
@@ -61,7 +63,7 @@ class PostBulkCreator
private
def create_row locale, actor, attributes, index
def create_row locale, actor, attributes, index, event, event_mutex
preflight =
PostCreatePreflight.new(
attributes: attributes,
@@ -74,9 +76,16 @@ class PostBulkCreator
existing_post: preflight[:existing_post] }
end
post = PostCreator.new(
actor: actor,
attributes: normalised_attributes(attributes, preflight, index)).create!(locale)
create_post = lambda do
VersionRecorder.with_event(event_type: 'posts.create', created_by_user: actor, event:) do
PostCreator.new(
actor: actor,
attributes: normalised_attributes(attributes, preflight, index)).create!(locale)
end
end
# Wait for the first event transaction before other workers reuse its ID.
committed = event_mutex.synchronize { event.persisted? }
post = committed ? create_post.call : event_mutex.synchronize(&create_post)
result = {
status: 'created',
post: { id: post.id } }
+2 -1
ファイルの表示
@@ -18,7 +18,8 @@ class PostCreator
original_created_from: @attributes[:original_created_from].presence,
original_created_before: @attributes[:original_created_before].presence)
ApplicationRecord.transaction do
VersionRecorder.record_operation!(
event_type: 'posts.create', created_by_user: @actor) do
post.save!
post.thumbnail.attach(thumbnail_attachment) if thumbnail_attachment.present?
snapshot_tags = planned_snapshot_tags(locale)
+21 -33
ファイルの表示
@@ -1,42 +1,30 @@
# Shared literal parsing and reference resolution for current and snapshot post search.
class PostTagSearch
def self.parse_references(raw, literals)
return nil unless raw.key?('tag_refs')
references = QueryState.records(raw, 'tag_refs')
valid = references.is_a?(Array) && references.all? do |reference|
next false unless reference.is_a?(Hash) && reference['name'].is_a?(String)
ids = [reference['tag_id'], reference['external_tag_id']].compact
((ids.any? || reference['missing'] == true) &&
ids.all? { _1.is_a?(Integer) && _1.positive? } &&
(!reference.key?('missing') || (reference['missing'] == true && ids.empty?)) &&
(!reference.key?('negative') || [true, false].include?(reference['negative'])))
end
raise ArgumentError unless valid
references.map { _1['name'] } == literals ? references : nil
end
def self.literals(locale, raw, references: nil)
raw.split.each_with_index.filter_map do |token, index|
next if token.downcase == 'type:or'
reference = references&.[](index)
name = token.sub(/\Anot:/i, '')
name = TagName.canonicalise(locale, name).first unless reference || locale.nil?
negative = reference&.key?('negative') ? reference['negative'] :
token.downcase.start_with?('not:')
{ name:, negative:, reference: }
def self.literals(locale, raw)
tokens = raw.split.reject { _1.downcase == 'type:or' }
names = tokens.map { _1.sub(/\Anot:/i, '') }
registered = TagName.where(language_code: locale.language_code, name: names)
.pluck(:name, :tag_id).to_h
primary = TagName.where(language_code: locale.language_code,
tag_id: registered.values, primary_flg: true).pluck(:tag_id, :name).to_h
tokens.zip(names).map do |token, name|
{ name: primary[registered[name]] || name,
negative: token.downcase.start_with?('not:') }
end
end
def self.references(locale, literal)
reference = literal[:reference]
return [reference['tag_id'], reference['external_tag_id']] if reference
references_for(locale, [literal]).fetch(literal[:name])
end
[TagName.where(language_code: locale.language_code, name: literal[:name])
.distinct.pluck(:tag_id),
ExternalTag.where("CONCAT(platform, ':', name) = ?", literal[:name]).pluck(:id)]
def self.references_for(locale, literals)
names = literals.map { _1[:name] }.uniq
internal = TagName.where(language_code: locale.language_code, name: names)
.pluck(:name, :tag_id).group_by(&:first)
external = names.empty? ? {} : ExternalTag.where(
"CONCAT(platform, ':', name) IN (?)", names)
.pluck(Arel.sql("CONCAT(platform, ':', name)"), :id).group_by(&:first)
names.to_h { |name| [name, [internal.fetch(name, []).map(&:last).uniq,
external.fetch(name, []).map(&:last).uniq]] }
end
end
+8 -5
ファイルの表示
@@ -1,16 +1,19 @@
class PostVersionRecorder < VersionRecorder
def self.record! post:, event_type:, created_by_user:
new(post:, event_type:, created_by_user:).record!
def self.record! post:, event_type:, created_by_user:, event: nil
new(post:, event_type:, event:, created_by_user:).record!
end
def initialize post:, event_type:, created_by_user:
super(record: post, event_type:, created_by_user:)
def initialize post:, event_type:, created_by_user:, event: nil
super(record: post, event_type:, event:, created_by_user:)
end
def self.ensure_snapshot! post, created_by_user:
return if post.post_versions.exists?
record!(post:, event_type: :create, created_by_user:)
event = current_event || Event.new(event_type: 'posts.create',
created_by_user:,
created_at: post.created_at)
record!(post:, event_type: :create, event:, created_by_user:)
end
private
+5 -5
ファイルの表示
@@ -1,10 +1,9 @@
# Public query grammar shared with frontend/src/lib/queryState.ts.
class QueryState
SCHEMAS = {
'conditions' => ['field', 'operator', 'value', 'end', 'blank', 'input', 'values'],
'conditions' => ['field', 'operator', 'value', 'end', 'blank', 'input', 'values', 'grain', 'value_type'],
'condition_values' => ['condition', 'value', 'value_type'],
'drill' => ['field', 'grain', 'value', 'value_type'],
'tag_refs' => ['name', 'tag_id', 'external_tag_id', 'negative', 'missing'],
'column_order' => ['value', 'value_type'],
'measures' => ['key', 'axis', 'value', 'value_type'] }.freeze
@@ -41,11 +40,11 @@ class QueryState
_root, index, field = name.split('.')
record = records[index.to_i] ||= {}
if ['blank', 'negative', 'missing', 'values'].include?(field)
if ['blank', 'values'].include?(field)
raise ArgumentError unless ['0', '1'].include?(value)
value = value == '1'
elsif ['tag_id', 'external_tag_id', 'axis', 'condition'].include?(field)
elsif ['axis', 'condition'].include?(field)
raise ArgumentError unless value.match?(/\A[0-9]{1,18}\z/)
value = value.to_i
@@ -57,7 +56,8 @@ class QueryState
records.sort.map do |_index, record|
type = record.delete('value_type')
if ['drill', 'measures', 'column_order', 'condition_values'].include?(key)
if (['drill', 'measures', 'column_order', 'condition_values'].include?(key) ||
(key == 'conditions' && record['operator'] == 'axis_eq'))
raise ArgumentError unless ['null', 'string', 'number', 'boolean'].include?(type)
raise ArgumentError unless type == 'null' || record.key?('value')
+6 -2
ファイルの表示
@@ -4,7 +4,8 @@ class TagDescriptionOperation
end
def self.create! locale, tag:, body:, created_by_user:, message: nil
ApplicationRecord.transaction do
VersionRecorder.record_operation!(
event_type: 'tag_descriptions.create', created_by_user:, reason: message) do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
tag.lock!
raise ActiveRecord::RecordNotUnique if TagDescription.exists?(
@@ -19,7 +20,9 @@ class TagDescriptionOperation
end
def self.update! locale, tag:, body:, updated_by_user:, message: nil, base_revision_id: nil
ApplicationRecord.transaction do
VersionRecorder.record_operation!(
event_type: 'tag_descriptions.update',
created_by_user: updated_by_user, reason: message) do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
tag.lock!
description = TagDescription.lock.find_by!(
@@ -43,6 +46,7 @@ class TagDescriptionOperation
description.created_by_user_id = user.id if creating
description.update!(body:, version_no:, updated_by_user_id: user.id)
TagDescriptionVersion.create!(
event: VersionRecorder.current_event,
tag_id: description.tag_id, language_code: description.language_code,
version_no:, event_type: creating ? 'create' : 'update', body:,
source_language_code: description.source_language_code,
+6 -3
ファイルの表示
@@ -3,7 +3,8 @@ class TagGroupOperation
def self.create!(locale, name:, description:, members:, placements:, actor:,
require_compatibility: false)
ApplicationRecord.transaction do
VersionRecorder.record_operation!(
event_type: 'tag_groups.create', created_by_user: actor) do
group = TagGroup.new(version_no: 1, created_by_user: actor,
updated_by_user: actor)
attributes = prepare!(locale:, group:, name:, description:, members:, placements:)
@@ -19,7 +20,8 @@ class TagGroupOperation
def self.update!(locale, group:, name:, description:, members:, placements:, actor:,
require_compatibility: false)
ApplicationRecord.transaction do
VersionRecorder.record_operation!(
event_type: 'tag_groups.update', created_by_user: actor) do
group.lock!
attributes = prepare!(locale:, group:, name:, description:, members:, placements:)
if unchanged?(group, attributes)
@@ -307,7 +309,8 @@ class TagGroupOperation
private_class_method :persist_placements!
def self.persist_version!(group:, attributes:, actor:, event_type:)
TagGroupVersion.create!(tag_group: group, version_no: group.version_no,
TagGroupVersion.create!(event: VersionRecorder.current_event,
tag_group: group, version_no: group.version_no,
event_type:, name: group.name,
description: group.description,
members: attributes.fetch(:members),
+4 -4
ファイルの表示
@@ -1,13 +1,13 @@
class TagVersionRecorder < VersionRecorder
def self.record! tag:, event_type:, created_by_user:, **_options
def self.record! tag:, event_type:, created_by_user:, event: nil, **_options
Tag.transaction do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
new(tag:, event_type:, created_by_user:).record!
new(tag:, event_type:, event:, created_by_user:).record!
end
end
def initialize tag:, event_type:, created_by_user:
super(record: tag, event_type:, created_by_user:)
def initialize tag:, event_type:, created_by_user:, event: nil
super(record: tag, event_type:, event:, created_by_user:)
end
private
+26 -3
ファイルの表示
@@ -1,9 +1,29 @@
class VersionRecorder
EVENT_TYPES = ['create', 'update', 'discard', 'restore'].freeze
def initialize record:, event_type:, created_by_user:
def self.current_event
ActiveSupport::IsolatedExecutionState[:version_event]
end
def self.with_event event_type:, created_by_user:, reason: nil, event: nil
previous = current_event
ActiveSupport::IsolatedExecutionState[:version_event] =
previous || event || Event.new(event_type:, created_by_user:, reason:)
yield
ensure
ActiveSupport::IsolatedExecutionState[:version_event] = previous
end
def self.record_operation! event_type:, created_by_user:, reason: nil, savepoint_flg: false
with_event(event_type:, created_by_user:, reason:) do
ApplicationRecord.transaction(requires_new: savepoint_flg) { yield }
end
end
def initialize record:, event_type:, created_by_user:, event: nil
@record = record
@event_type = event_type.to_s
@event = event
@created_by_user = created_by_user
validate_event_type!
@@ -42,8 +62,11 @@ class VersionRecorder
def base_attributes latest
{ version_no: (latest&.version_no || 0) + 1,
event_type: @event_type,
created_at: Time.current,
created_by_user: @created_by_user }
event: @event || self.class.current_event || Event.new(
event_type: "#{ record_class.table_name }.#{ @event_type }",
created_by_user: @created_by_user),
created_by_user: @created_by_user,
created_at: Time.current }
end
def update_record_version_no! version_no
+50 -46
ファイルの表示
@@ -9,12 +9,14 @@ module Youtube
end
def sync!
video_ids = discover_video_ids
return if video_ids.empty?
VersionRecorder.with_event(event_type: 'posts.sync', created_by_user: nil) do
video_ids = discover_video_ids
return if video_ids.empty?
video_ids.each_slice(50) do |ids|
@client.videos(ids).fetch('items', []).each do |item|
sync_video!(VideoItem.new(item))
video_ids.each_slice(50) do |ids|
@client.videos(ids).fetch('items', []).each do |item|
sync_video!(VideoItem.new(item))
end
end
end
end
@@ -46,59 +48,61 @@ module Youtube
end
def sync_video! video
post = Post.where('url REGEXP ?', youtube_url_regexp(video.id)).first
ApplicationRecord.transaction do
post = Post.where('url REGEXP ?', youtube_url_regexp(video.id)).first
original_created_from = video.published_at.change(sec: 0)
original_created_before = original_created_from + 1.minute
original_created_from = video.published_at.change(sec: 0)
original_created_before = original_created_from + 1.minute
post_created = false
post_changed = false
post_created = false
post_changed = false
if post
post.assign_attributes(title: video.title,
original_created_from:,
original_created_before:,
thumbnail_base: video.thumbnail_url)
if post
post.assign_attributes(title: video.title,
original_created_from:,
original_created_before:,
thumbnail_base: video.thumbnail_url)
post_changed = post.changed?
post.save! if post_changed
post_changed = post.changed?
post.save! if post_changed
attach_thumbnail_if_needed!(post, video.thumbnail_url)
else
post_created = true
post = Post.create!(
title: video.title,
url: video.url,
thumbnail_base: video.thumbnail_url,
uploaded_user_id: nil,
original_created_from:,
original_created_before:)
attach_thumbnail_if_needed!(post, video.thumbnail_url)
else
post_created = true
post = Post.create!(
title: video.title,
url: video.url,
thumbnail_base: video.thumbnail_url,
uploaded_user_id: nil,
original_created_from:,
original_created_before:)
attach_thumbnail_if_needed!(post, video.thumbnail_url)
attach_thumbnail_if_needed!(post, video.thumbnail_url)
sync_post_tags!(post, [Tag.tagme.id, Tag.bot.id, Tag.youtube.id, Tag.video.id])
end
sync_post_tags!(post, [Tag.tagme.id, Tag.bot.id, Tag.youtube.id, Tag.video.id])
end
kept_tag_ids = post.tags.pluck(:id).to_set
desired_tag_ids = kept_tag_ids.to_a
kept_tag_ids = post.tags.pluck(:id).to_set
desired_tag_ids = kept_tag_ids.to_a
deerjikist = Deerjikist.find_by(platform: :youtube, code: video.channel_id)
if deerjikist
desired_tag_ids.delete(Tag.no_deerjikist.id)
desired_tag_ids << deerjikist.tag_id
elsif post.tags.where(category: :deerjikist).none?
desired_tag_ids << Tag.no_deerjikist.id
end
deerjikist = Deerjikist.find_by(platform: :youtube, code: video.channel_id)
if deerjikist
desired_tag_ids.delete(Tag.no_deerjikist.id)
desired_tag_ids << deerjikist.tag_id
elsif post.tags.where(category: :deerjikist).none?
desired_tag_ids << Tag.no_deerjikist.id
end
desired_tag_ids.uniq!
desired_tag_ids.uniq!
sync_post_tags!(post, desired_tag_ids, current_tag_ids: kept_tag_ids)
sync_post_tags!(post, desired_tag_ids, current_tag_ids: kept_tag_ids)
if post_created
PostVersionRecorder.record!(post:, event_type: :create, created_by_user: nil)
elsif post_changed || kept_tag_ids != desired_tag_ids.to_set
PostVersionRecorder.ensure_snapshot!(post, created_by_user: nil)
PostVersionRecorder.record!(post:, event_type: :update, created_by_user: nil)
if post_created
PostVersionRecorder.record!(post:, event_type: :create, created_by_user: nil)
elsif post_changed || kept_tag_ids != desired_tag_ids.to_set
PostVersionRecorder.ensure_snapshot!(post, created_by_user: nil)
PostVersionRecorder.record!(post:, event_type: :update, created_by_user: nil)
end
end
end
+1
ファイルの表示
@@ -3,6 +3,7 @@ Rails.application.routes.draw do
get 'counts/definitions/:resource/candidates', to: 'counts_tables#candidates'
get 'counts/query', to: 'counts_tables#index'
post 'counts/export/jobs', to: 'counts_exports#create_job'
post 'counts/export/jobs/:token/download', to: 'counts_exports#start_download'
get 'counts/export/jobs/:token', to: 'counts_exports#job_status'
delete 'counts/export/jobs/:token', to: 'counts_exports#cancel_job'
post 'counts/export/prepare', to: 'counts_exports#prepare'
+5
ファイルの表示
@@ -0,0 +1,5 @@
class AddLinkedTagIdsToNicoTagVersions < ActiveRecord::Migration[8.0]
def change
add_column :nico_tag_versions, :linked_tag_ids, :text
end
end
+42
ファイルの表示
@@ -0,0 +1,42 @@
class MigrateLinkedTagsInNicoTagVersions < ActiveRecord::Migration[8.0]
class MigrationTagVersion < ActiveRecord::Base
self.table_name = 'tag_versions'
end
class MigrationNicoTagVersion < ActiveRecord::Base
self.table_name = 'nico_tag_versions'
end
def up
# 除票外部タグの恢復はないので除票履歴があれば全消しでよいはず.
execute <<~SQL
DELETE
t1
FROM
nico_tag_versions t1
INNER JOIN
nico_tag_versions t2
ON
t1.tag_id = t2.tag_id
AND t2.event_type = 'discard'
SQL
MigrationNicoTagVersion.find_each do |ntv|
linked_tag_ids = ntv.linked_tags.split.map { |name|
# 厳密には一意ぢゃないけどめんだぅだからこれでいいや.
MigrationTagVersion
.order(created_at: :desc, id: :desc)
.find_by("names ->> '$.ja.name' = ?", name)
&.tag_id
}.compact.uniq.sort.join(' ')
ntv.update_columns(linked_tag_ids:)
end
change_column_null :nico_tag_versions, :linked_tag_ids, false
end
def down
raise ActiveRecord::IrreversibleMigration, '戻せません.'
end
end
+6
ファイルの表示
@@ -0,0 +1,6 @@
class RemoveLinkedTagsFromNicoTagVersions < ActiveRecord::Migration[8.0]
def change
remove_column :nico_tag_versions, :linked_tags
remove_column :nico_tag_versions, :name
end
end
+20
ファイルの表示
@@ -0,0 +1,20 @@
class CreateEvents < ActiveRecord::Migration[8.0]
def change
create_table :events do |t|
t.string :event_type, null: false, index: true
t.text :reason
t.references :reverted_from_event, foreign_key: { to_table: :events }
t.json :metadata
t.binary :ip_address, limit: 16, index: true
t.references :created_by_user, index: true, foreign_key: false
t.datetime :created_at, null: false, index: true
end
add_reference :post_versions, :event, foreign_key: true, after: :original_created_before
add_reference :tag_versions, :event, foreign_key: true, after: :deprecated_at
add_reference :nico_tag_versions, :event, foreign_key: true, after: :event_type
add_reference :material_versions, :event, foreign_key: true
add_reference :tag_description_versions, :event, foreign_key: true, after: :source_version_no
add_reference :tag_group_versions, :event, foreign_key: true, after: :placements
end
end
+239
ファイルの表示
@@ -0,0 +1,239 @@
class BackfillEventsFromExistingVersions < ActiveRecord::Migration[8.0]
class MigrationEvent < ActiveRecord::Base
self.table_name = 'events'
end
class MigrationPostVersion < ActiveRecord::Base
self.table_name = 'post_versions'
end
class MigrationTagVersion < ActiveRecord::Base
self.table_name = 'tag_versions'
end
class MigrationNicoTagVersion < ActiveRecord::Base
self.table_name = 'nico_tag_versions'
end
class MigrationMaterialVersion < ActiveRecord::Base
self.table_name = 'material_versions'
end
class MigrationTagDescriptionVersion < ActiveRecord::Base
self.table_name = 'tag_description_versions'
end
class MigrationTagGroupVersion < ActiveRecord::Base
self.table_name = 'tag_group_versions'
end
class MigrationLocale < ActiveRecord::Base
self.table_name = 'locales'
end
def up
# create → posts.create
# update → posts.update
if MigrationPostVersion.where.not(event_type: ['create', 'update']).exists?
raise
end
# create → materials.create
# update → materials.update
if MigrationMaterialVersion.where.not(event_type: ['create', 'update']).exists?
raise
end
# create → posts.create / posts.update / materials.create / materials.update
# / external_tags.update / tags.update / unknown
# update → tags.update / locales.create
# discard → unknown
if MigrationTagVersion.where.not(event_type: ['create', 'update', 'discard']).exists?
raise
end
# create → posts.create / posts.update / unknown
# update → external_tags.update
if MigrationNicoTagVersion.where.not(event_type: ['create', 'update']).exists?
raise
end
# create → tag_descriptions.create
# update → tag_descriptions.update
if MigrationTagDescriptionVersion.where.not(event_type: ['create', 'update']).exists?
raise
end
# create → tag_groups.create
# update → tag_groups.update
if MigrationTagGroupVersion.where.not(event_type: ['create', 'update']).exists?
raise
end
# post_versions の Backfill
MigrationPostVersion.find_each do |pv|
ev = MigrationEvent.create!(event_type: "posts.#{ pv.event_type }",
created_by_user_id: pv.created_by_user_id,
created_at: pv.created_at)
pv.update_columns(event_id: ev.id)
end
# material_versions の Backfill
MigrationMaterialVersion.find_each do |m|
ev = MigrationEvent.create!(event_type: "materials.#{ m.event_type }",
created_by_user_id: m.created_by_user_id,
created_at: m.created_at)
m.update_columns(event_id: ev.id)
end
# ロカール追加に伴ふ該当語言のタグ名称追加異動のみ分離
ids_cause_of_locale_creation = []
MigrationLocale.where('created_at > ?', MigrationLocale.minimum(:created_at)).find_each do |l|
ev = MigrationEvent.create!(event_type: 'locales.create',
created_by_user_id: nil,
created_at: l.created_at)
MigrationTagVersion
.where('created_at > ?', l.created_at)
.where(event_type: 'update', event_id: nil)
.find_each do |t|
next unless t.names.key?(l.language_code)
prev = MigrationTagVersion.find_by(tag_id: t.tag_id, version_no: t.version_no - 1)
next if prev.nil? || prev.names.key?(l.language_code)
t.update_columns(event_id: ev.id)
ids_cause_of_locale_creation << t.id
end
ev.destroy! unless MigrationTagVersion.where(event_id: ev.id).exists?
end
# nico_tag_versions の Backfill
MigrationNicoTagVersion.find_each do |et|
ev =
case et.event_type
when 'create'
post =
MigrationPostVersion
.where('JSON_CONTAINS(tags_json, ?) <> 0', { external_tag_id: et.tag_id }.to_json)
.order(:created_at, :id)
.first
if post
MigrationEvent.find(post.event_id)
else
create_unknown_event!(et)
end
when 'update'
MigrationEvent.create!(event_type: 'external_tags.update',
created_by_user_id: et.created_by_user_id,
created_at: et.created_at)
end
et.update_columns(event_id: ev.id)
end
# tag_versions の Backfill(異動および除票)
MigrationTagVersion
.where(event_type: ['update', 'discard'])
.where.not(id: ids_cause_of_locale_creation)
.find_each do |t|
ev =
case t.event_type
when 'update'
MigrationEvent.create!(event_type: 'tags.update',
created_by_user_id: t.created_by_user_id,
created_at: t.created_at)
when 'discard'
create_unknown_event!(t)
end
t.update_columns(event_id: ev.id)
end
# tag_versions の Backfill(起票のみ)
MigrationTagVersion.where(event_type: 'create').find_each do |t|
post =
MigrationPostVersion
.where('JSON_CONTAINS(tags_json, ?) <> 0', { tag_id: t.tag_id }.to_json)
.order(:created_at, :id)
.first
material = MigrationMaterialVersion.where(tag_id: t.tag_id).order(:created_at, :id).first
external_tag =
MigrationNicoTagVersion
.where(event_type: 'update')
.where("CONCAT(' ', linked_tag_ids, ' ') LIKE ?", "% #{ t.tag_id } %")
.order(:created_at, :id)
.first
tag =
MigrationTagVersion
.where(event_type: 'update')
.where("CONCAT(' ', parent_tag_ids, ' ') LIKE ?", "% #{ t.tag_id } %")
.order(:created_at, :id)
.first
target = nil
if post
target = post
end
if material && (!(target) || material.created_at < target.created_at)
target = material
end
if external_tag && (!(target) || external_tag.created_at < target.created_at)
target = external_tag
end
if tag && (!(target) || tag.created_at < target.created_at)
target = tag
end
ev =
if target
MigrationEvent.find(target.event_id)
else
create_unknown_event!(t)
end
t.update_columns(event_id: ev.id)
end
# tag_description_versions の Backfill
MigrationTagDescriptionVersion.find_each do |td|
ev = MigrationEvent.create!(event_type: "tag_descriptions.#{ td.event_type }",
created_by_user_id: td.created_by_user_id,
created_at: td.created_at)
td.update_columns(event_id: ev.id)
end
# tag_group_versions の Backfill
MigrationTagGroupVersion.find_each do |tg|
ev = MigrationEvent.create!(event_type: "tag_groups.#{ tg.event_type }",
created_by_user_id: tg.created_by_user_id,
created_at: tg.created_at)
tg.update_columns(event_id: ev.id)
end
change_column_null :post_versions, :event_id, false
change_column_null :tag_versions, :event_id, false
change_column_null :nico_tag_versions, :event_id, false
change_column_null :material_versions, :event_id, false
change_column_null :tag_description_versions, :event_id, false
change_column_null :tag_group_versions, :event_id, false
end
def down
raise ActiveRecord::IrreversibleMigration, '戻せません.'
end
def create_unknown_event! source
MigrationEvent.create!(event_type: 'unknown',
created_by_user_id: source.created_by_user_id,
created_at: source.created_at)
end
end
生成ファイル
+37 -3
ファイルの表示
@@ -10,7 +10,7 @@
#
# It's strongly recommended that you check this file into your version control system.
ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
ActiveRecord::Schema[8.0].define(version: 2026_11_17_010000) do
create_table "active_storage_attachments", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.string "name", null: false
t.string "record_type", null: false
@@ -48,6 +48,21 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
t.index ["tag_id"], name: "index_deerjikists_on_tag_id"
end
create_table "events", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.string "event_type", null: false
t.text "reason"
t.bigint "reverted_from_event_id"
t.json "metadata"
t.binary "ip_address", limit: 16
t.bigint "created_by_user_id"
t.datetime "created_at", null: false
t.index ["created_at"], name: "index_events_on_created_at"
t.index ["created_by_user_id"], name: "index_events_on_created_by_user_id"
t.index ["event_type"], name: "index_events_on_event_type"
t.index ["ip_address"], name: "index_events_on_ip_address"
t.index ["reverted_from_event_id"], name: "index_events_on_reverted_from_event_id"
end
create_table "external_tags", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.string "platform", limit: 16, null: false
t.string "name", null: false
@@ -250,8 +265,10 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
t.string "source_path"
t.string "source_file_id"
t.string "normalized_source_key"
t.bigint "event_id", null: false
t.index ["created_by_user_id"], name: "index_material_versions_on_created_by_user_id"
t.index ["discarded_at"], name: "index_material_versions_on_discarded_at"
t.index ["event_id"], name: "index_material_versions_on_event_id"
t.index ["file_blob_id"], name: "index_material_versions_on_file_blob_id"
t.index ["material_id", "version_no"], name: "index_material_versions_on_material_id_and_version_no", unique: true
t.index ["material_id"], name: "index_material_versions_on_material_id"
@@ -301,12 +318,13 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
t.bigint "tag_id", null: false
t.integer "version_no", null: false
t.string "event_type", null: false
t.string "name", null: false
t.text "linked_tags", null: false
t.bigint "event_id", null: false
t.datetime "created_at", null: false
t.bigint "created_by_user_id"
t.text "linked_tag_ids", null: false
t.index ["created_at"], name: "index_nico_tag_versions_on_created_at"
t.index ["created_by_user_id", "created_at"], name: "index_nico_tag_versions_on_created_by_user_id_and_created_at", order: { created_at: :desc }
t.index ["event_id"], name: "index_nico_tag_versions_on_event_id"
t.index ["tag_id", "created_at"], name: "index_nico_tag_versions_on_tag_id_and_created_at", order: { created_at: :desc }
t.index ["tag_id", "version_no"], name: "index_nico_tag_versions_on_tag_id_and_version_no", unique: true
t.check_constraint "`version_no` > 0", name: "nico_tag_versions_version_no_positive"
@@ -380,10 +398,12 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
t.text "parent_post_ids", null: false
t.datetime "original_created_from"
t.datetime "original_created_before"
t.bigint "event_id", null: false
t.datetime "created_at", null: false
t.bigint "created_by_user_id"
t.integer "video_ms"
t.index ["created_by_user_id"], name: "index_post_versions_on_created_by_user_id"
t.index ["event_id"], name: "index_post_versions_on_event_id"
t.index ["post_id", "version_no"], name: "index_post_versions_on_post_id_and_version_no", unique: true
t.index ["post_id"], name: "index_post_versions_on_post_id"
t.index ["video_ms", "post_id"], name: "idx_post_versions_video_ms_post_id"
@@ -448,9 +468,11 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
t.text "body", null: false
t.string "source_language_code", limit: 16
t.integer "source_version_no"
t.bigint "event_id", null: false
t.bigint "created_by_user_id"
t.datetime "created_at", null: false
t.index ["created_by_user_id"], name: "index_tag_description_versions_on_created_by_user_id"
t.index ["event_id"], name: "index_tag_description_versions_on_event_id"
t.index ["tag_id", "language_code", "version_no"], name: "idx_on_tag_id_language_code_version_no_ced77e57e6", unique: true
t.index ["tag_id", "source_language_code", "source_version_no"], name: "fk_rails_4f13094dbc"
t.index ["tag_id"], name: "index_tag_description_versions_on_tag_id"
@@ -533,9 +555,11 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
t.text "description"
t.json "members", null: false
t.json "placements", null: false
t.bigint "event_id", null: false
t.bigint "created_by_user_id"
t.datetime "created_at", null: false
t.index ["created_by_user_id"], name: "index_tag_group_versions_on_created_by_user_id"
t.index ["event_id"], name: "index_tag_group_versions_on_event_id"
t.index ["tag_group_id", "version_no"], name: "index_tag_group_versions_on_tag_group_id_and_version_no", unique: true
t.index ["tag_group_id"], name: "index_tag_group_versions_on_tag_group_id"
t.check_constraint "`version_no` > 0", name: "chk_tag_group_versions_version_no_positive"
@@ -611,11 +635,13 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
t.text "aliases", null: false
t.text "parent_tag_ids", null: false
t.datetime "deprecated_at"
t.bigint "event_id", null: false
t.datetime "created_at", null: false
t.bigint "created_by_user_id"
t.json "names", null: false
t.index ["created_at"], name: "index_tag_versions_on_created_at"
t.index ["created_by_user_id", "created_at"], name: "index_tag_versions_on_created_by_user_id_and_created_at", order: { created_at: :desc }
t.index ["event_id"], name: "index_tag_versions_on_event_id"
t.index ["tag_id", "created_at"], name: "index_tag_versions_on_tag_id_and_created_at", order: { created_at: :desc }
t.index ["tag_id", "version_no"], name: "index_tag_versions_on_tag_id_and_version_no", unique: true
t.check_constraint "`version_no` > 0", name: "tag_versions_version_no_positive"
@@ -630,6 +656,7 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
t.integer "post_count", default: 0, null: false
t.integer "version_no", null: false
t.index ["deprecated_at"], name: "index_tags_on_deprecated_at"
t.index ["post_count"], name: "index_tags_on_post_count"
t.check_constraint "(`deprecated_at` is null) or (`category` <> _utf8mb4'nico')", name: "chk_tags_deprecated_at_not_nico"
t.check_constraint "`version_no` > 0", name: "chk_tags_version_no_positive"
end
@@ -771,6 +798,7 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
add_foreign_key "active_storage_attachments", "active_storage_blobs", column: "blob_id"
add_foreign_key "active_storage_variant_records", "active_storage_blobs", column: "blob_id"
add_foreign_key "events", "events", column: "reverted_from_event_id"
add_foreign_key "gekanator_ai_runs", "gekanator_question_suggestions"
add_foreign_key "gekanator_games", "posts", column: "correct_post_id"
add_foreign_key "gekanator_games", "posts", column: "guessed_post_id"
@@ -792,6 +820,7 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
add_foreign_key "material_sync_sources", "users", column: "created_by_user_id"
add_foreign_key "material_sync_sources", "users", column: "updated_by_user_id"
add_foreign_key "material_sync_suppressions", "users", column: "created_by_user_id"
add_foreign_key "material_versions", "events"
add_foreign_key "material_versions", "materials"
add_foreign_key "material_versions", "materials", column: "parent_id"
add_foreign_key "material_versions", "users", column: "created_by_user_id"
@@ -802,6 +831,7 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
add_foreign_key "materials", "users", column: "updated_by_user_id"
add_foreign_key "nico_tag_relations", "external_tags", column: "nico_tag_id"
add_foreign_key "nico_tag_relations", "tags"
add_foreign_key "nico_tag_versions", "events"
add_foreign_key "nico_tag_versions", "users", column: "created_by_user_id"
add_foreign_key "post_external_tags", "external_tags"
add_foreign_key "post_external_tags", "posts"
@@ -813,10 +843,12 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
add_foreign_key "post_tags", "posts"
add_foreign_key "post_tags", "tags"
add_foreign_key "post_tags", "users", column: "created_user_id"
add_foreign_key "post_versions", "events"
add_foreign_key "post_versions", "posts"
add_foreign_key "post_versions", "users", column: "created_by_user_id"
add_foreign_key "posts", "users", column: "uploaded_user_id"
add_foreign_key "settings", "users"
add_foreign_key "tag_description_versions", "events"
add_foreign_key "tag_description_versions", "tag_description_versions", column: ["tag_id", "source_language_code", "source_version_no"], primary_key: ["tag_id", "language_code", "version_no"]
add_foreign_key "tag_descriptions", "languages", column: "language_code", primary_key: "code"
add_foreign_key "tag_descriptions", "tag_description_versions", column: ["tag_id", "source_language_code", "source_version_no"], primary_key: ["tag_id", "language_code", "version_no"]
@@ -835,6 +867,7 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
add_foreign_key "tag_group_tags", "tag_groups"
add_foreign_key "tag_group_tags", "tags"
add_foreign_key "tag_group_tags", "users", column: "created_by_user_id"
add_foreign_key "tag_group_versions", "events"
add_foreign_key "tag_groups", "users", column: "created_by_user_id"
add_foreign_key "tag_groups", "users", column: "updated_by_user_id"
add_foreign_key "tag_implications", "tags"
@@ -844,6 +877,7 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
add_foreign_key "tag_names", "tags"
add_foreign_key "tag_similarities", "tags"
add_foreign_key "tag_similarities", "tags", column: "target_tag_id"
add_foreign_key "tag_versions", "events"
add_foreign_key "tag_versions", "users", column: "created_by_user_id"
add_foreign_key "theatre_comments", "theatres"
add_foreign_key "theatre_comments", "users"
+15 -2
ファイルの表示
@@ -9,8 +9,21 @@
# end
ApplicationRecord.transaction do
Language.find_or_create_by!(code: 'ain') { _1.name = 'ain' }
Script.find_or_create_by!(code: 'Kana') { _1.name = 'Kana' }
Language.find_or_create_by!(code: 'ja') { _1.name = '日本語' }
Script.find_or_create_by!(code: 'Jpan') { _1.name = '漢字、仮名文字およびラテン文字' }
Region.find_or_create_by!(code: 392) do
_1.alpha_code = 'JP'
_1.name = '日本国'
end
Locale.find_or_create_by!(code: 'ja') do
_1.language_code = 'ja'
_1.script_code = 'Jpan'
_1.region_code = 392
_1.name = '日本語'
end
Language.find_or_create_by!(code: 'ain') { _1.name = 'ヤウンモシルン アイヌ イタㇰ' }
Script.find_or_create_by!(code: 'Kana') { _1.name = 'カナモジ オヨビ ラテンモジ' }
end
post_url_sanitisation_rules = [
+118 -114
ファイルの表示
@@ -67,129 +67,133 @@ namespace :nico do
end
data = JSON.parse(stdout)
data.each do |datum|
code = datum['code']
post =
Post
.where('url REGEXP ?', "nicovideo\\.jp/watch/#{ Regexp.escape(code) }([^0-9]|$)")
.first
VersionRecorder.with_event(event_type: 'posts.sync', created_by_user: nil) do
data.each do |datum|
ApplicationRecord.transaction do
code = datum['code']
post =
Post
.where('url REGEXP ?', "nicovideo\\.jp/watch/#{ Regexp.escape(code) }([^0-9]|$)")
.first
title = datum['title']
original_created_at = datum['uploaded_at'] &&
Time.strptime(datum['uploaded_at'], '%Y-%m-%d %H:%M:%S')
original_created_from = original_created_at&.change(sec: 0)
original_created_before = original_created_from&.+(1.minute)
title = datum['title']
original_created_at = datum['uploaded_at'] &&
Time.strptime(datum['uploaded_at'], '%Y-%m-%d %H:%M:%S')
original_created_from = original_created_at&.change(sec: 0)
original_created_before = original_created_from&.+(1.minute)
post_created = false
post_changed = false
post_created = false
post_changed = false
if post
attrs = { title:, original_created_from:, original_created_before: }
if post
attrs = { title:, original_created_from:, original_created_before: }
unless post.thumbnail.attached?
thumbnail_base = fetch_thumbnail.(post.url) rescue nil
if thumbnail_base.present?
attrs[:thumbnail_base] = thumbnail_base
begin
post.attach_thumbnail_from_url!(thumbnail_base)
rescue Post::RemoteThumbnailFetchFailed => e
payload = {
post_id: post.id,
thumbnail_base:,
error: e.class.name,
message: e.message }
Rails.logger.info("nico_sync_thumbnail_fetch_failed #{ payload.to_json }")
unless post.thumbnail.attached?
thumbnail_base = fetch_thumbnail.(post.url) rescue nil
if thumbnail_base.present?
attrs[:thumbnail_base] = thumbnail_base
begin
post.attach_thumbnail_from_url!(thumbnail_base)
rescue Post::RemoteThumbnailFetchFailed => e
payload = {
post_id: post.id,
thumbnail_base:,
error: e.class.name,
message: e.message }
Rails.logger.info("nico_sync_thumbnail_fetch_failed #{ payload.to_json }")
end
end
end
post.assign_attributes(attrs)
post_changed = post.changed?
if post_changed
post.save!
end
else
post_created = true
url = "https://www.nicovideo.jp/watch/#{ code }"
thumbnail_base = fetch_thumbnail.(url) rescue nil
post = Post.new(title:, url:, thumbnail_base:, uploaded_user: nil,
original_created_from:, original_created_before:)
if thumbnail_base.present?
begin
post.attach_thumbnail_from_url!(thumbnail_base)
rescue Post::RemoteThumbnailFetchFailed => e
payload = {
post_id: nil,
thumbnail_base:,
error: e.class.name,
message: e.message }
Rails.logger.info("nico_sync_thumbnail_fetch_failed #{ payload.to_json }")
end
end
post.save!
sync_post_tags!(post, [Tag.tagme.id, Tag.bot.id, Tag.niconico.id, Tag.video.id])
end
# 既存のタグ Id. 集合
kept_tag_ids = post.tags.pluck(:id).to_set
# 既存の外部タグ Id. 集合
kept_external_tag_ids = post.external_tags.nico.pluck(:id).to_set
# 記載すべき外部タグ Id. のリスト
desired_external_tag_ids = []
# 記載すべき内部タグ Id. のリスト
desired_tag_ids = kept_tag_ids.to_a
datum['tags'].each do |raw|
name = TagNameSanitisationRule.sanitise("nico:#{ raw }").delete_prefix('nico:')
tag = ExternalTag.find_or_create_by!(platform: :nico, name:)
unless tag.nico_tag_versions.exists?
NicoTagVersionRecorder.record!(external_tag: tag,
event_type: :create,
created_by_user: nil)
end
desired_external_tag_ids << tag.id
# 新たに記載される外部タグと連携される内部タグを記載
# 連携タグは記載すれども消除せず.
unless tag.id.in?(kept_external_tag_ids)
desired_tag_ids.concat(tag.linked_tags.pluck(:id))
end
end
end
post.assign_attributes(attrs)
post_changed = post.changed?
if post_changed
post.save!
end
else
post_created = true
url = "https://www.nicovideo.jp/watch/#{ code }"
thumbnail_base = fetch_thumbnail.(url) rescue nil
post = Post.new(title:, url:, thumbnail_base:, uploaded_user: nil,
original_created_from:, original_created_before:)
if thumbnail_base.present?
begin
post.attach_thumbnail_from_url!(thumbnail_base)
rescue Post::RemoteThumbnailFetchFailed => e
payload = {
post_id: nil,
thumbnail_base:,
error: e.class.name,
message: e.message }
Rails.logger.info("nico_sync_thumbnail_fetch_failed #{ payload.to_json }")
deerjikist = Deerjikist.find_by(platform: :nico, code: datum['user'])
if deerjikist
desired_tag_ids << deerjikist.tag_id
elsif !(Tag.where(id: kept_tag_ids).where(category: :deerjikist).exists?)
desired_tag_ids << Tag.no_deerjikist.id
end
desired_external_tag_ids.uniq!
desired_tag_ids.uniq!
# 外部タグの記載に際しては “bot 操作” タグを記載しなぃ.
if kept_tag_ids != desired_tag_ids.to_set
desired_tag_ids << Tag.bot.id
desired_tag_ids.uniq!
end
tags_changed =
kept_tag_ids != desired_tag_ids.to_set ||
kept_external_tag_ids != desired_external_tag_ids.to_set
sync_post_tags!(post, desired_tag_ids, current_tag_ids: kept_tag_ids)
sync_post_external_tags!(post, desired_external_tag_ids,
current_external_tag_ids: kept_external_tag_ids)
if post_created
PostVersionRecorder.record!(post:, event_type: :create, created_by_user: nil)
elsif post_changed || tags_changed
PostVersionRecorder.ensure_snapshot!(post, created_by_user: nil)
PostVersionRecorder.record!(post:, event_type: :update, created_by_user: nil)
end
end
post.save!
sync_post_tags!(post, [Tag.tagme.id, Tag.bot.id, Tag.niconico.id, Tag.video.id])
end
# 既存のタグ Id. 集合
kept_tag_ids = post.tags.pluck(:id).to_set
# 既存の外部タグ Id. 集合
kept_external_tag_ids = post.external_tags.nico.pluck(:id).to_set
# 記載すべき外部タグ Id. のリスト
desired_external_tag_ids = []
# 記載すべき内部タグ Id. のリスト
desired_tag_ids = kept_tag_ids.to_a
datum['tags'].each do |raw|
name = TagNameSanitisationRule.sanitise("nico:#{ raw }").delete_prefix('nico:')
tag = ExternalTag.find_or_create_by!(platform: :nico, name:)
unless tag.nico_tag_versions.exists?
NicoTagVersionRecorder.record!(external_tag: tag,
event_type: :create,
created_by_user: nil)
end
desired_external_tag_ids << tag.id
# 新たに記載される外部タグと連携される内部タグを記載
# 連携タグは記載すれども消除せず.
unless tag.id.in?(kept_external_tag_ids)
desired_tag_ids.concat(tag.linked_tags.pluck(:id))
end
end
deerjikist = Deerjikist.find_by(platform: :nico, code: datum['user'])
if deerjikist
desired_tag_ids << deerjikist.tag_id
elsif !(Tag.where(id: kept_tag_ids).where(category: :deerjikist).exists?)
desired_tag_ids << Tag.no_deerjikist.id
end
desired_external_tag_ids.uniq!
desired_tag_ids.uniq!
# 外部タグの記載に際しては “bot 操作” タグを記載しなぃ.
if kept_tag_ids != desired_tag_ids.to_set
desired_tag_ids << Tag.bot.id
desired_tag_ids.uniq!
end
tags_changed =
kept_tag_ids != desired_tag_ids.to_set ||
kept_external_tag_ids != desired_external_tag_ids.to_set
sync_post_tags!(post, desired_tag_ids, current_tag_ids: kept_tag_ids)
sync_post_external_tags!(post, desired_external_tag_ids,
current_external_tag_ids: kept_external_tag_ids)
if post_created
PostVersionRecorder.record!(post:, event_type: :create, created_by_user: nil)
elsif post_changed || tags_changed
PostVersionRecorder.ensure_snapshot!(post, created_by_user: nil)
PostVersionRecorder.record!(post:, event_type: :update, created_by_user: nil)
end
end
end
+1
ファイルの表示
@@ -102,6 +102,7 @@ RSpec.describe 'Tag group persistence constraints' do
6.times { nodes = [{ 'tag_id' => tag.id, 'children' => nodes }] }
reject_statement do
insert_group_row('tag_group_versions', {
event_id: create_event!(event_type: 'unknown').id,
tag_group_id: group_id, version_no: 2, event_type: 'update', name: '制約確認',
members: [{ 'tag_id' => tag.id, 'version_no' => tag.version_no }].to_json,
placements: nodes.to_json, created_at: Time.current })
+1
ファイルの表示
@@ -10,6 +10,7 @@ RSpec.describe 'TagVersion names database constraint' do
def insert_snapshot!(snapshot)
TagVersion.insert_all!([{ tag_id: tag.id, version_no: 1, event_type: 'create',
event_id: create_event!(event_type: 'unknown').id,
name: 'snapshot_name', category: 'general',
aliases: '', parent_tag_ids: '', names: snapshot,
created_at: Time.current }])
+43 -8
ファイルの表示
@@ -11,28 +11,63 @@ RSpec.describe ExternalTag, type: :model do
expect(external.reload.name).to eq('raw tag[]')
end
it 'deletes post associations without deleting posts or version history' do
it 'does not allow a persisted external tag to change its platform or name' do
external = create(:external_tag, name: 'immutable')
expect { external.update!(name: 'renamed') }
.to raise_error(ActiveRecord::RecordInvalid)
expect(external.errors[:name]).to include('は変更できません.')
external.reload
external.platform = nil
expect { external.save! }.to raise_error(ActiveRecord::RecordInvalid)
expect(external.errors[:platform]).to include('は変更できません.')
expect(external.reload).to have_attributes(platform: 'nico', name: 'immutable')
end
it 'rejects delete, which bypasses destroy callbacks' do
external = create(:external_tag)
expect { external.delete }.to raise_error(ActiveRecord::ReadOnlyRecord)
expect(external.reload).to be_persisted
end
it 'increments the counter through post associations without changing identity' do
external = create(:external_tag, name: 'counter')
post = create(:post)
expect {
PostExternalTag.create!(post:, external_tag: external)
}.to change { external.reload.post_count }.by(1)
expect(external.reload).to have_attributes(platform: 'nico', name: 'counter')
end
it 'does not destroy external tags or their post associations and history' do
external = create(:external_tag)
post = create(:post)
PostExternalTag.create!(post:, external_tag: external)
version = NicoTagVersionRecorder.record!(
external_tag: external, event_type: :create, created_by_user: nil)
external.destroy!
expect { external.destroy! }.to raise_error(ActiveRecord::RecordNotDestroyed)
expect(PostExternalTag.where(external_tag_id: external.id)).to be_empty
expect(post.reload.external_tags).to be_empty
expect(external.reload).to be_persisted
expect(PostExternalTag.exists?(post:, external_tag: external)).to be(true)
expect(post.reload.external_tags).to contain_exactly(external)
expect(version.reload.tag_id).to eq(external.id)
end
it 'deletes external links without deleting linked internal tags' do
it 'does not destroy an external tag or its links to internal tags' do
external = create(:external_tag)
tag = create(:tag)
NicoTagRelation.create!(nico_tag: external, tag:)
relation = NicoTagRelation.create!(nico_tag: external, tag:)
external.destroy!
expect { external.destroy! }.to raise_error(ActiveRecord::RecordNotDestroyed)
expect(NicoTagRelation.where(nico_tag_id: external.id)).to be_empty
expect(external.reload).to be_persisted
expect(relation.reload).to be_persisted
expect(Tag.exists?(tag.id)).to be(true)
end
end
+146
ファイルの表示
@@ -0,0 +1,146 @@
require 'rails_helper'
RSpec.describe HistoryAccess do
include_context 'Ainu locale'
let!(:tag) { create(:tag) }
it 'uses the complete tag-description identity and version order at equal timestamps' do
locale
rows = ['ja', 'ain'].flat_map do |language|
['create', 'discard', 'restore'].each_with_index.map do |event, index|
TagDescriptionVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag_id: tag.id, language_code: language,
version_no: index + 1, event_type: language == 'ain' ? 'update' : event,
body: "#{ language } #{ index }", created_at: Time.utc(2024))
end
end
described_class.set(user: nil) do
expect(described_class.visible(TagDescriptionVersion.where(id: rows.map(&:id))).pluck(:id))
.to match_array([rows[2], *rows.last(3)].map(&:id))
expect { described_class.require!(rows.first) }.to raise_error(HistoryAccess::Forbidden)
end
described_class.set(user: create(:user, :admin)) do
expect(described_class.visible(TagDescriptionVersion.where(id: rows.map(&:id))).count).to eq(6)
end
end
it 'invalidates public artifacts after a new discard and binds admin artifacts to their actor' do
public_metadata = described_class.set(user: nil) { described_class.authority }
admin = create(:user, :admin)
admin_metadata = described_class.set(user: admin) { described_class.authority }
TagVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag:, version_no: 1, event_type: 'discard', name: 'private',
category: 'general',
names: { ja: { name: 'private', script_code: 'Jpan',
auto_generated: false, aliases: [] } },
aliases: '', parent_tag_ids: '', created_at: Time.utc(2024))
described_class.set(user: nil) do
expect(described_class.authorised_artifact?(public_metadata)).to be(false)
expect(described_class.authorised_artifact?(admin_metadata)).to be(false)
end
described_class.set(user: create(:user, :admin)) do
expect(described_class.authorised_artifact?(admin_metadata)).to be(false)
end
described_class.set(user: admin) do
expect(described_class.authorised_artifact?(admin_metadata)).to be(true)
admin.update!(role: :guest)
expect(described_class.authorised_artifact?(admin_metadata)).to be(false)
end
end
it 'denies a blob attached to a discarded post including its derived attachment' do
post = create(:post)
blob = ActiveStorage::Blob.create_before_direct_upload!(filename: 'private.png',
byte_size: 1, checksum: Base64.strict_encode64('checksum'), content_type: 'image/png')
ActiveStorage::Attachment.create!(name: 'thumbnail', record: post, blob:)
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post:, version_no: 1, event_type: 'discard', url: post.url,
title: 'private', tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024))
variant = ActiveStorage::VariantRecord.create!(blob:, variation_digest: 'spec-variation')
derived = ActiveStorage::Blob.create_before_direct_upload!(filename: 'derived.png',
byte_size: 1, checksum: Base64.strict_encode64('checksum'), content_type: 'image/png')
ActiveStorage::Attachment.create!(name: 'image', record: variant, blob: derived)
described_class.set(user: nil) do
expect(described_class.blob_visible?(blob)).to be(false)
expect(described_class.blob_visible?(derived)).to be(false)
end
described_class.set(user: create(:user, :admin)) do
expect(described_class.blob_visible?(derived)).to be(true)
end
end
end
RSpec.describe 'Counts artifact policy and external references' do
it 'rejects files created before the masked-summary policy' do
HistoryAccess.set(user: nil) do
metadata = HistoryAccess.authority.except('artifact_policy')
expect(HistoryAccess.authorised_artifact?(metadata)).to be(false)
end
end
it 'does not expose a private linked tag ID through an external history snapshot' do
tag = create(:tag, primary_name: 'private-linked-name')
TagVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag:, version_no: 1, event_type: 'discard', name: 'private-linked-name',
category: 'general', aliases: '', parent_tag_ids: '', created_at: Time.utc(2024),
names: { ja: { name: 'private-linked-name', script_code: 'Jpan',
auto_generated: false, aliases: [] } })
external = create(:external_tag, name: 'public-external')
row = NicoTagVersion.create!(external_tag: external, version_no: 1, event_type: 'create',
event_id: create_event!(event_type: 'unknown').id,
linked_tag_ids: tag.id.to_s,
created_at: Time.utc(2024))
post = create(:post)
snapshot = PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post:, version_no: 1, event_type: 'create',
title: 'external reference', url: post.url, tags: '',
tags_json: [{ external_tag_id: external.id }], parent_post_ids: '',
created_at: Time.utc(2024, 2))
HistoryAccess.set(user: nil) do
expect(HistoryAccess.visible(NicoTagVersion.where(id: row.id))).to be_empty
expect(HistoryAccess.visible(PostVersion.where(id: snapshot.id))).to be_empty
end
HistoryAccess.set(user: create(:user, :admin)) do
expect(HistoryAccess.visible(NicoTagVersion.where(id: row.id))).to exist
end
public_row = NicoTagVersion.create!(external_tag: external, version_no: 2, event_type: 'update',
event_id: create_event!(event_type: 'unknown').id,
linked_tag_ids: '',
created_at: Time.utc(2024, 7))
HistoryAccess.set(user: nil) do
expect(HistoryAccess.visible(NicoTagVersion.where(id: public_row.id))).to exist
expect(HistoryAccess.visible(PostVersion.where(id: snapshot.id))).to be_empty
end
end
it 'propagates the recorded tag version policy to its post snapshot' do
parent = create(:tag, primary_name: 'private parent')
child = create(:tag, primary_name: 'referencing child')
[[parent, 'discard', '', 'private parent'],
[child, 'create', parent.id.to_s, 'referencing child']].each do |tag, event, parents, name|
TagVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag:, version_no: 1, event_type: event, name:,
category: 'general', aliases: '', parent_tag_ids: parents, created_at: Time.utc(2023),
names: { ja: { name:, script_code: 'Jpan',
auto_generated: false, aliases: [] } })
end
post = create(:post)
row = PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post:, version_no: 1, event_type: 'create', title: 'snapshot',
url: post.url, tags: '', parent_post_ids: '',
tags_json: [{ tag_id: child.id, version_no: 1, name: 'referencing child', sections: [] }],
created_at: Time.utc(2024))
HistoryAccess.set(user: nil) do
expect(HistoryAccess.visible(PostVersion.where(id: row.id))).to be_empty
end
end
end
+24 -1
ファイルの表示
@@ -23,9 +23,22 @@ RSpec.describe 'Locale activation', type: :model do
.and_raise('backfill failed')
end
it 'groups all tag histories from locale creation into one event' do
previous_ids = TagVersion.pluck(:id)
expect { build_locale.save! }.to change { event_rows.length }.by(1)
event = event_rows.last
expect(event).to include('event_type' => 'locales.create', 'created_by_user_id' => nil)
versions = TagVersion.where.not(id: previous_ids)
expect(versions.pluck(:tag_id)).to include(first.id, second.id)
expect(versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
end
it 'rolls back a newly created locale and the preceding generated names' do
locale = build_locale
fail_second_generation!
events = event_rows
expect { locale.save! }.to raise_error('backfill failed')
@@ -33,6 +46,7 @@ RSpec.describe 'Locale activation', type: :model do
expect(generated_names).to be_empty
expect(first.reload.name('ja')).to eq('activation_first')
expect(second.reload.name('ja')).to eq('activation_second')
expect(event_rows).to eq(events)
end
it 'does not backfill while the locale is inactive' do
@@ -45,7 +59,14 @@ RSpec.describe 'Locale activation', type: :model do
locale = build_locale(deprecated_at: Time.current)
locale.save!
locale.update!(deprecated_at: nil)
previous_ids = TagVersion.pluck(:id)
expect { locale.update!(deprecated_at: nil) }.to change { event_rows.length }.by(1)
event = event_rows.last
expect(event).to include('event_type' => 'locales.restore', 'created_by_user_id' => nil)
versions = TagVersion.where.not(id: previous_ids)
expect(versions.pluck(:tag_id)).to include(first.id, second.id)
expect(versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
expect(generated_names.pluck(:tag_id)).to contain_exactly(first.id, second.id)
expect(generated_names).to all(have_attributes(
@@ -57,11 +78,13 @@ RSpec.describe 'Locale activation', type: :model do
locale.save!
previous_deprecated_at = locale.reload.deprecated_at
fail_second_generation!
events = event_rows
expect { locale.update!(deprecated_at: nil) }.to raise_error('backfill failed')
expect(locale.reload.deprecated_at).to eq(previous_deprecated_at)
expect(generated_names).to be_empty
expect(event_rows).to eq(events)
end
it 'preserves a human primary through deactivation and repeated activation' do
+14 -14
ファイルの表示
@@ -12,20 +12,20 @@ RSpec.describe PostVersion, type: :model do
let!(:post_version) do
PostVersion.create!(
post: post_record,
version_no: 1,
event_type: 'create',
title: post_record.title,
url: post_record.url,
thumbnail_base: post_record.thumbnail_base,
tags: post_record.snapshot_tag_names.join(' '),
tags_json: post_record.snapshot_tags_json,
parent_post_ids: post_record.snapshot_parent_post_ids.join(' '),
original_created_from: post_record.original_created_from,
original_created_before: post_record.original_created_before,
created_at: Time.current,
created_by_user: nil
)
event_id: create_event!(event_type: 'unknown').id,
post: post_record,
version_no: 1,
event_type: 'create',
title: post_record.title,
url: post_record.url,
thumbnail_base: post_record.thumbnail_base,
tags: post_record.snapshot_tag_names.join(' '),
tags_json: post_record.snapshot_tags_json,
parent_post_ids: post_record.snapshot_parent_post_ids.join(' '),
original_created_from: post_record.original_created_from,
original_created_before: post_record.original_created_before,
created_at: Time.current,
created_by_user: nil)
end
it 'is read only after create' do
+3 -4
ファイルの表示
@@ -10,23 +10,22 @@ RSpec.describe 'Shared Tag and ExternalTag IDs', type: :model do
expect(next_tag.id).to be > external.id
end
it 'does not reuse physically deleted IDs in either table' do
it 'does not reuse IDs after internal deletion while external tags remain' do
tag = create(:tag)
external = create(:external_tag)
TagName.where(tag_id: tag.id).delete_all
tag.destroy!
external.destroy!
expect(ExternalTag.exists?(external.id)).to be(true)
expect(create(:tag).id).to be > external.id
expect(create(:external_tag).id).to be > external.id
end
it 'retains existing overlapping IDs without renumbering on update or insert' do
it 'retains overlapping IDs after a tag update and a new allocation' do
tag = create(:tag)
external = create(:external_tag)
external.update_columns(id: tag.id)
tag.update!(category: :meta)
external.update!(name: 'updated_external')
before_tag = tag.reload.attributes
before_external = external.reload.attributes
+27
ファイルの表示
@@ -34,6 +34,31 @@ RSpec.describe TagNameSanitisationRule, type: :model do
described_class.create!(priority: 10, source_pattern: 'foo_', replacement: 'foo')
end
it 'groups renames, merges and affected post histories into one event' do
target = create(:tag, primary_name: 'foobar')
source = create(:tag, primary_name: 'temporary_source')
renamed = create(:tag, primary_name: 'temporary_rename')
source.tag_name('ja').update_columns(name: 'foo_bar')
renamed.tag_name('ja').update_columns(name: 'foo_baz')
post = create(:post)
PostTag.create!(post:, tag: source)
previous_tag_ids = TagVersion.pluck(:id)
previous_post_ids = PostVersion.pluck(:id)
expect { described_class.apply! }.to change { event_rows.length }.by(1)
event = event_rows.last
expect(event).to include(
'event_type' => 'tag_name_sanitisation_rules.apply', 'created_by_user_id' => nil)
tag_versions = TagVersion.where.not(id: previous_tag_ids)
expect(tag_versions.pluck(:tag_id).uniq).to contain_exactly(target.id, source.id, renamed.id)
expect(tag_versions.pluck(:event_type)).to include('discard', 'update')
expect(tag_versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
post_versions = PostVersion.where.not(id: previous_post_ids)
expect(post_versions.pluck(:post_id).uniq).to eq([post.id])
expect(post_versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
end
[:primary, :alias].each do |source_kind|
it "keeps the primary identity when a #{ source_kind } collides within its Tag" do
tag = create(:tag, primary_name: source_kind == :primary ? 'temporary' : 'foobar')
@@ -83,6 +108,7 @@ RSpec.describe TagNameSanitisationRule, type: :model do
PostTag.create!(post:, tag: source)
models = [Tag, TagName, TagVersion, TagDescription, TagDescriptionVersion, PostTag]
snapshots = models.map { |model| model.order(*Array(model.primary_key)).map(&:attributes) }
events = event_rows
expect { described_class.apply! }.to raise_error(ActiveRecord::RecordInvalid)
@@ -90,6 +116,7 @@ RSpec.describe TagNameSanitisationRule, type: :model do
model.order(*Array(model.primary_key)).map(&:attributes)
end
expect(after_state).to eq(snapshots)
expect(event_rows).to eq(events)
expect(post.reload.tags).to contain_exactly(source)
expect(target.reload.name('ja')).to eq('foobar')
end
+14 -14
ファイルの表示
@@ -665,20 +665,20 @@ RSpec.describe Tag, type: :model do
def create_post_version_for!(post, version_no: 1, event_type: 'create', created_by_user: nil)
PostVersion.create!(
post: post,
version_no: version_no,
event_type: event_type,
title: post.title,
url: post.url,
thumbnail_base: post.thumbnail_base,
tags: snapshot_tags(post),
tags_json: post.snapshot_tags_json,
parent_post_ids: post.snapshot_parent_post_ids.join(' '),
original_created_from: post.original_created_from,
original_created_before: post.original_created_before,
created_at: Time.current,
created_by_user: created_by_user
)
event_id: create_event!(event_type: 'unknown').id,
post: post,
version_no: version_no,
event_type: event_type,
title: post.title,
url: post.url,
thumbnail_base: post.thumbnail_base,
tags: snapshot_tags(post),
tags_json: post.snapshot_tags_json,
parent_post_ids: post.snapshot_parent_post_ids.join(' '),
original_created_from: post.original_created_from,
original_created_before: post.original_created_before,
created_at: Time.current,
created_by_user: created_by_user)
end
context 'when post versions are enabled' do
+51 -39
ファイルの表示
@@ -6,17 +6,18 @@ RSpec.describe VersionRecord, type: :model do
it 'makes TagVersion read only after create' do
version = TagVersion.create!(
tag: tag,
version_no: 1,
event_type: 'create',
name: primary_tag_name_for(tag, 'ja').name,
category: tag.category,
aliases: '',
names: { ja: { name: primary_tag_name_for(tag, 'ja').name,
script_code: 'Jpan', auto_generated: false, aliases: [] } },
parent_tag_ids: '',
created_at: Time.current,
created_by_user: nil)
event_id: create_event!(event_type: 'unknown').id,
tag: tag,
version_no: 1,
event_type: 'create',
name: primary_tag_name_for(tag, 'ja').name,
category: tag.category,
aliases: '',
names: { ja: { name: primary_tag_name_for(tag, 'ja').name,
script_code: 'Jpan', auto_generated: false, aliases: [] } },
parent_tag_ids: '',
created_at: Time.current,
created_by_user: nil)
expect {
version.update!(name: 'changed')
@@ -25,17 +26,18 @@ RSpec.describe VersionRecord, type: :model do
it 'prevents TagVersion destroy' do
version = TagVersion.create!(
tag: tag,
version_no: 1,
event_type: 'create',
name: primary_tag_name_for(tag, 'ja').name,
category: tag.category,
aliases: '',
names: { ja: { name: primary_tag_name_for(tag, 'ja').name,
script_code: 'Jpan', auto_generated: false, aliases: [] } },
parent_tag_ids: '',
created_at: Time.current,
created_by_user: nil)
event_id: create_event!(event_type: 'unknown').id,
tag: tag,
version_no: 1,
event_type: 'create',
name: primary_tag_name_for(tag, 'ja').name,
category: tag.category,
aliases: '',
names: { ja: { name: primary_tag_name_for(tag, 'ja').name,
script_code: 'Jpan', auto_generated: false, aliases: [] } },
parent_tag_ids: '',
created_at: Time.current,
created_by_user: nil)
expect {
version.destroy!
@@ -44,30 +46,40 @@ RSpec.describe VersionRecord, type: :model do
it 'makes NicoTagVersion read only after create' do
version = NicoTagVersion.create!(
external_tag: nico_tag,
version_no: 1,
event_type: 'create',
name: "nico:#{ nico_tag.name }",
linked_tags: '',
created_at: Time.current,
created_by_user: nil
)
event_id: create_event!(event_type: 'external_tags.create').id,
external_tag: nico_tag,
version_no: 1,
event_type: 'create',
linked_tag_ids: '',
created_at: Time.current,
created_by_user: nil)
expect {
version.update!(name: 'nico:changed')
version.update!(linked_tag_ids: '123')
}.to raise_error(ActiveRecord::ReadOnlyRecord)
end
it 'rejects discard and restore versions for external tags' do
[:discard, :restore].each do |event_type|
version = NicoTagVersion.new(
external_tag: nico_tag, version_no: 1, event_type:,
linked_tag_ids: '', created_at: Time.current)
expect(version).not_to be_valid
expect(version.errors[:event_type]).not_to be_empty
end
end
it 'prevents NicoTagVersion destroy' do
version = NicoTagVersion.create!(
external_tag: nico_tag,
version_no: 1,
event_type: 'create',
name: "nico:#{ nico_tag.name }",
linked_tags: '',
created_at: Time.current,
created_by_user: nil
)
event_id: create_event!(event_type: 'external_tags.create').id,
external_tag: nico_tag,
version_no: 1,
event_type: 'create',
linked_tag_ids: '',
created_at: Time.current,
created_by_user: nil)
expect {
version.destroy!
+1
ファイルの表示
@@ -44,6 +44,7 @@ RSpec.configure do |config|
end
config.include TestRecords
config.include EventRecords
config.include RakeTaskHelper
# FactoryBot の create / create_list を使へるやぅにする
+126
ファイルの表示
@@ -0,0 +1,126 @@
require 'rails_helper'
require 'tmpdir'
RSpec.describe 'Counts native GET acknowledgement', type: :request do
include ActiveSupport::Testing::TimeHelpers
around do |example|
Dir.mktmpdir('counts-download-request') do |path|
@counts_download_dir = Pathname.new(path)
example.run
end
end
before do
allow(CountsExportDownload).to receive(:directory).and_return(@counts_download_dir)
end
def ready_job(format, payload: 'payload')
CountsExportJobs.pulse
post '/counts/export/jobs', params: { resource: 'posts', definition: 'history',
mode: 'details', columns: 'resource_id', locale: 'ja', format: }
expect(response).to have_http_status(:accepted)
job = response.parsed_body.fetch('job')
signature = CountsExportJobs.verifier.verified(job, purpose: 'counts_export_job')
owner = signature.fetch('owner')
exporter = instance_double(CountsExporter, resource: 'posts', format:, body: [payload])
file = HistoryAccess.set(user: nil) { CountsExportDownload.prepare(exporter, owner) }
state = CountsExportJobs.read(signature.fetch('id'))
state.merge!('status' => 'ready', 'token' => file)
CountsExportJobs.write(state)
post "/counts/export/jobs/#{ job }/download"
expect(response).to have_http_status(:ok)
[job, file, response.parsed_body.fetch('download_attempt')]
end
['csv', 'json'].each do |format|
it "delivers every byte of a multi-chunk #{ format } file through HTTP" do
sign_in_as(create(:user, :member))
rows = Array.new(5000) { |index| { id: index, name: '日本語,名称' } }
payload = if format == 'json'
rows.to_json
else
"id,name\r\n" + rows.map { CSV.generate_line(_1.values) }.join
end
job, file, attempt = ready_job(format, payload:)
owner = CountsExportJobs.verifier.verified(job, purpose: 'counts_export_job').fetch('owner')
metadata = HistoryAccess.set(user: nil) { CountsExportDownload.find(file, owner) }
expected = File.binread(metadata.fetch('path'))
expect(expected.bytesize).to be > 64 * 1024
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:ok)
expect(response.body.bytesize).to eq(expected.bytesize)
expect(response.body.b).to eq(expected)
expect(response.body.b).to eq(payload.b)
get "/counts/export/jobs/#{ job }"
expect(response.parsed_body['download_status']).to eq('started')
end
it "acknowledges only an authorised #{ format } GET body, not HEAD or acceptance" do
job, file, attempt = ready_job(format)
get "/counts/export/jobs/#{ job }"
expect(response.parsed_body['download_status']).to eq('pending')
head "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:method_not_allowed)
get "/counts/export/jobs/#{ job }"
expect(response.parsed_body['download_status']).to eq('pending')
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:ok)
expect(response.headers['Content-Disposition']).to include("counts-posts.#{ format }")
expect(response.body).to eq('payload')
get "/counts/export/jobs/#{ job }"
expect(response.parsed_body['download_status']).to eq('started')
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:not_found)
end
end
it 'rejects wrong attempts, cancellation and a newly discarded resource' do
job, file, attempt = ready_job('json')
get "/counts/export/files/#{ file }", params: { job:, attempt: 'wrong' }
expect(response).to have_http_status(:not_found)
delete "/counts/export/jobs/#{ job }"
expect(response).to have_http_status(:ok)
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:not_found)
job, file, attempt = ready_job('json')
record = create(:post)
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post: record, version_no: 1, event_type: 'discard', title: 'private',
url: record.url, tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024))
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:not_found)
expect(response.body).not_to include('payload', 'private')
end
it 'does not let a second browser session inspect the job or download its file' do
job, file, attempt = ready_job('json')
other = ActionDispatch::Integration::Session.new(Rails.application)
other.get("/counts/export/jobs/#{ job }")
expect(other.response).to have_http_status(:not_found)
other.get("/counts/export/files/#{ file }", params: { job:, attempt: })
expect(other.response).to have_http_status(:not_found)
expect(other.response.body).not_to include('payload')
end
it 'does not acknowledge an expired attempt' do
job, file, attempt = ready_job('json')
travel_to(Time.current + 46.seconds) do
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:not_found)
get "/counts/export/jobs/#{ job }"
expect(response.parsed_body['download_status']).to eq('timeout')
end
end
it 'records a missing file without accepting the GET as started' do
job, file, attempt = ready_job('json')
owner = CountsExportJobs.verifier.verified(job, purpose: 'counts_export_job').fetch('owner')
metadata = HistoryAccess.set(user: nil) { CountsExportDownload.find(file, owner) }
File.unlink(metadata.fetch('path'))
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:not_found)
get "/counts/export/jobs/#{ job }"
expect(response.parsed_body['download_status']).to eq('unavailable')
end
end
+42
ファイルの表示
@@ -0,0 +1,42 @@
require 'rails_helper'
RSpec.describe 'History boundaries shared with Counts', type: :request do
let!(:tag) { create(:tag, primary_name: 'description tag') }
let!(:description) do
TagDescription.create!(tag:, language_code: 'ja', version_no: 2, body: 'private body')
end
before do
['create', 'discard'].each_with_index do |event, index|
TagDescriptionVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag_id: tag.id, language_code: 'ja',
version_no: index + 1, event_type: event, body: 'private body',
created_at: Time.utc(2024))
end
end
it 'denies the same discarded description through Counts, the facade and the ordinary API' do
get '/counts/query', params: { resource: 'tag_descriptions', definition: 'history',
mode: 'details', locale: 'ja', execution_at: '2025-01-01T00:00:00Z' }
expect(response).to have_http_status(:ok)
expect(response.parsed_body).to include('count' => 0, 'output_count' => 2,
'excluded_count' => 2, 'versions' => [])
["/wiki/#{ description.id }", "/tags/#{ tag.id }/description"].each do |path|
get path, params: { locale: 'ja' }
expect(response).to have_http_status(:not_found)
expect(response.body).not_to include('private body')
end
get '/sitemap_entries/descriptions', params: { locale: 'ja' }
expect(response).to have_http_status(:ok)
expect(response.parsed_body['ids']).not_to include(description.id)
end
it 'permits the server-authenticated administrator to read the discarded description' do
sign_in_as(create(:user, :admin))
get "/tags/#{ tag.id }/description", params: { locale: 'ja' }
expect(response).to have_http_status(:ok)
expect(response.parsed_body['body']).to eq('private body')
expect(response.headers['Cache-Control']).to include('no-store')
end
end
+196
ファイルの表示
@@ -0,0 +1,196 @@
require 'rails_helper'
RSpec.describe 'Counts masked search', type: :request do
let!(:tag) { create(:tag, primary_name: '秘密のタグ') }
let(:actor) { create(:user, :member) }
let(:admin) { create(:user, :admin) }
let(:base) do
{ 'resource' => 'tags', 'definition' => 'history', 'mode' => 'summary',
'locale' => 'ja', 'execution_at' => '2025-01-01T00:00:00Z',
'timezone' => 'UTC', 'rows' => '', 'cols' => '', 'sort' => '' }
end
before do
[1, 2].each do |number|
TagVersion.create!(tag:, version_no: number,
event_type: number == 1 ? 'create' : 'discard', name: '秘密のタグ',
category: 'general', aliases: '秘密の別名', parent_tag_ids: '',
names: { ja: { name: '秘密のタグ', script_code: 'Jpan', auto_generated: false,
aliases: [{ name: '秘密の別名', script_code: 'Jpan' }] } },
created_by_user: actor, created_at: Time.utc(2023, number))
end
end
def query(extra = {})
get '/counts/query', params: base.merge(extra)
expect(response).to have_http_status(:ok), extra.inspect
response.parsed_body
end
def predicate(field, operator, value, index = 0)
prefix = "conditions.#{ index }"
result = { 'conditions' => '', "#{ prefix }.field" => field,
"#{ prefix }.operator" => operator }
if ['in', 'any', 'all', 'none', 'not_in'].include?(operator)
result.merge!("#{ prefix }.values" => '1', 'condition_values' => '',
"condition_values.#{ index }.condition" => index.to_s,
"condition_values.#{ index }.value_type" => 'string',
"condition_values.#{ index }.value" => value)
else
result["#{ prefix }.value"] = value
end
result
end
it 'rejects every name and alias predicate on private history' do
['name', 'name_ja', 'names.name', 'aliases.name', 'names.aliases.name'].each do |field|
secret = field.include?('aliases') ? '秘密の別名' : '秘密のタグ'
multiple = !['name', 'name_ja'].include?(field)
operators = multiple ? ['any', 'all', 'none'] : ['in', 'not_in']
['eq', 'ne', 'like', *operators].each do |operator|
[secret, '********', '%', ''].each do |value|
expected = (value == '' && ['eq', 'ne', 'like'].include?(operator)) ? 2 : 0
expect(query(predicate(field, operator, value))['count']).to eq(expected)
end
end
blank = predicate(field, 'eq', '').merge('conditions.0.blank' => '1')
expect(query(blank)['count']).to eq(0)
end
end
it 'ignores missing and empty scalar values unless blank is checked' do
['name', 'name_ja', 'names.name', 'aliases.name'].each do |field|
['eq', 'ne', 'like'].each do |operator|
condition = predicate(field, operator, '')
[condition, condition.except('conditions.0.value')].each do |empty|
expect(query(empty)['count']).to eq(2)
expect(query(empty.merge('conditions.0.blank' => '0'))['count']).to eq(2)
expect(query(empty.merge('conditions.0.blank' => '1'))['count']).to eq(0)
end
end
end
end
it 'keeps details totals unchanged for inactive private predicates' do
settings = predicate('name', 'ne', '').merge('mode' => 'details')
result = query(settings)
expect(result['count']).to eq(0)
expect(result['versions']).to be_empty
expect(result['output_count']).to eq(2)
active = query(settings.merge('conditions.0.blank' => '1'))
expect(active['output_count']).to eq(0)
end
it 'ignores missing sets but preserves explicit empty set semantics' do
{ 'name' => ['in', 'not_in'], 'aliases.name' => ['any', 'all', 'none'] }.each do
|field, operators|
operators.each do |operator|
condition = { 'conditions' => '', 'conditions.0.field' => field,
'conditions.0.operator' => operator }
expect(query(condition)['count']).to eq(2)
empty = condition.merge('conditions.0.values' => '1', 'condition_values' => '')
expected = ['in', 'any'].include?(operator) ? 0 : 2
expect(query(empty)['count']).to eq(expected)
expect(query(condition.merge('conditions.0.blank' => '1'))['count']).to eq(0)
# An explicit empty string element is a real value, not an unspecified set.
expect(query(predicate(field, operator, ''))['count']).to eq(0)
end
end
end
it 'ignores empty private predicates regardless of their position' do
first = predicate('name', 'ne', '')
.merge(predicate('resource_id', 'eq', tag.id.to_s, 1))
last = predicate('resource_id', 'eq', tag.id.to_s)
.merge(predicate('aliases.name', 'like', '', 1))
expect(query(first)['count']).to eq(2)
expect(query(last)['count']).to eq(2)
active = predicate('name', 'ne', 'unrelated')
.merge(predicate('aliases.name', 'like', '', 1))
expect(query(active)['count']).to eq(0)
end
it 'searches public literal markers and treats axis predicates as explicit' do
public_tag = create(:tag, primary_name: '********')
TagVersion.create!(tag: public_tag, version_no: 1, event_type: 'create',
name: '********', category: 'general', aliases: '', parent_tag_ids: '',
names: { ja: { name: '********', script_code: 'Jpan', auto_generated: false,
aliases: [] } }, created_at: Time.utc(2023))
expect(query['count']).to eq(3)
['eq', 'ne', 'like', 'in', 'not_in'].each do |operator|
expected = ['ne', 'not_in'].include?(operator) ? 0 : 1
expect(query(predicate('name', operator, '********'))['count']).to eq(expected)
end
blank = predicate('aliases.name', 'eq', '').merge('conditions.0.blank' => '1')
expect(query(blank)['count']).to eq(1)
['********', '', nil].each do |value|
condition = { 'conditions' => '', 'conditions.0.field' => 'name',
'conditions.0.operator' => 'axis_eq',
'conditions.0.value_type' => value.nil? ? 'null' : 'string' }
condition['conditions.0.value'] = value unless value.nil?
expected = value == '********' ? 1 : 0
expect(query(condition)['count']).to eq(expected)
drill = { 'drill' => '', 'drill.0.field' => 'name',
'drill.0.value_type' => value.nil? ? 'null' : 'string' }
drill['drill.0.value'] = value unless value.nil?
expect(query(drill)['count']).to eq(expected)
end
end
it 'keeps public identity and timeline predicates searchable' do
expect(query['count']).to eq(2)
values = { 'resource_id' => tag.id.to_s, 'version_no' => '2',
'event_type' => 'discard', 'user' => actor.id.to_s,
'created_at' => '2023-02-01T00:00:00Z' }
values.each do |field, value|
expected = ['resource_id', 'user'].include?(field) ? 2 : 1
expect(query(predicate(field, 'eq', value))['count']).to eq(expected)
end
condition = predicate('name', 'ne', '秘密のタグ')
.merge(predicate('event_type', 'eq', 'discard', 1))
expect(query(condition)['count']).to eq(0)
expect(query(predicate('category', 'eq', '********'))['count']).to eq(0)
expect(query(predicate('category', 'eq', 'general'))['count']).to eq(0)
end
it 'keeps public predicates available after combining multiple conditions' do
condition = predicate('resource_id', 'eq', tag.id.to_s)
.merge(predicate('event_type', 'eq', 'discard', 1))
expect(query(condition)['count']).to eq(1)
condition.merge!(predicate('name', 'ne', 'unrelated', 2))
expect(query(condition)['count']).to eq(0)
end
it 'keeps real name searches for administrators and excludes private details' do
condition = predicate('name', 'eq', '秘密のタグ')
details = query(condition.merge('mode' => 'details'))
expect(details['versions']).to be_empty
expect(details['output_count']).to eq(0)
masked = query(predicate('name', 'eq', '********').merge('mode' => 'details'))
expect(masked['versions']).to be_empty
expect(masked['output_count']).to eq(0)
sign_in_as(admin)
expect(query(condition)['count']).to eq(2)
expect(query(condition.merge('mode' => 'details'))['versions'].length).to eq(2)
end
it 'rejects private names reached through a post tag snapshot' do
post = create(:post)
PostVersion.create!(post:, version_no: 1, event_type: 'create', title: 'post',
url: post.url, tags: '', parent_post_ids: '',
tags_json: [{ tag_id: tag.id, version_no: 1, name: '秘密のタグ',
category: 'general', sections: [] }],
created_at: Time.utc(2024))
['tags.name', 'tags.names.name', 'tags.name_ja'].each do |field|
['eq', 'ne', 'like', 'any', 'all', 'none'].each do |operator|
['秘密のタグ', '********', '%'].each do |value|
settings = predicate(field, operator, value).merge('resource' => 'posts')
expect(query(settings)['count']).to eq(0)
end
end
blank = predicate(field, 'eq', '').merge('resource' => 'posts',
'conditions.0.blank' => '1')
expect(query(blank)['count']).to eq(0)
end
end
end
+17 -12
ファイルの表示
@@ -1,6 +1,6 @@
require 'rails_helper'
RSpec.describe 'Counts API', type: :request do
RSpec.describe 'Legacy Counts API', type: :request do
let!(:actor) { create(:user, :member) }
let!(:post) { create(:post) }
let!(:tag) { create(:tag, primary_name: 'counts tag', category: :general) }
@@ -10,12 +10,16 @@ RSpec.describe 'Counts API', type: :request do
category: 'general', sections: [] }
[1, 2, 3].each do |number|
time = Time.utc(2024, number == 3 ? 2 : 1, number, 0, 0)
PostVersion.create!(post:, version_no: number,
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post:, version_no: number,
event_type: number == 1 ? 'create' : 'update',
tags: '', url: post.url, title: "Snapshot #{ number }", parent_post_ids: '',
tags_json: number == 1 ? [snapshot] : [],
created_by_user: number == 3 ? nil : actor, created_at: time)
TagVersion.create!(tag:, version_no: number,
TagVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag:, version_no: number,
event_type: number == 1 ? 'create' : 'update',
name: 'counts tag', category: 'general', aliases: '', parent_tag_ids: '',
names: { ja: { name: 'counts tag', script_code: 'Jpan',
@@ -28,16 +32,16 @@ RSpec.describe 'Counts API', type: :request do
context resource do
let(:path) { "/counts/#{ resource }" }
let(:id) { resource == 'posts' ? post.id : tag.id }
let(:filters) { { resource_id: id, user_id: actor.id,
let(:filters) { { resource_id: id, user_id: actor.id, latest: '0',
from: '2024-01-01T00:00Z', before: '2024-02-01T00:00Z' } }
it 'uses the same AND filters for SQL totals, groups and details' do
get path, params: filters.merge(axis: 'event_type')
get path, params: filters.merge(axis: 'month')
expect(response).to have_http_status(:ok)
summary = response.parsed_body
expect(summary['count']).to eq(2)
expect(summary['groups'].sum { _1['count'] }).to eq(2)
expect(summary['groups'].map { _1['key'] }).to eq(['create', 'update'])
expect(summary['groups'].map { _1['key'] }).to eq(['2024-01'])
get path, params: filters.merge(mode: 'details')
expect(response.parsed_body['count']).to eq(2)
@@ -45,16 +49,16 @@ RSpec.describe 'Counts API', type: :request do
end
it 'paginates details and groups independently from totals' do
get path, params: { resource_id: id, mode: 'details', limit: 1, page: 2 }
get path, params: { resource_id: id, latest: '0', mode: 'details', limit: 1, page: 2 }
expect(response.parsed_body['count']).to eq(3)
expect(response.parsed_body['versions'].sole['version_no']).to eq(2)
get path, params: { resource_id: id, axis: 'month', limit: 1, page: 2 }
get path, params: { resource_id: id, latest: '0', axis: 'month', limit: 1, page: 2 }
expect(response.parsed_body['group_count']).to eq(2)
expect(response.parsed_body['groups'].sole).to include('key' => '2024-02', 'count' => 1)
end
it 'groups null actors and supports their drill-down filter' do
get path, params: { resource_id: id, axis: 'user' }
get path, params: { resource_id: id, latest: '0', axis: 'user' }
expect(response.parsed_body['groups']).to include(
{ 'key' => nil, 'name' => nil, 'count' => 1 })
get path, params: { resource_id: id, mode: 'details', user_id: 'none' }
@@ -62,7 +66,7 @@ RSpec.describe 'Counts API', type: :request do
end
it 'uses an inclusive start and exclusive end' do
get path, params: { resource_id: id,
get path, params: { resource_id: id, latest: '0',
from: '2024-01-01T00:00Z', before: '2024-01-02T00:00Z' }
expect(response.parsed_body['count']).to eq(1)
end
@@ -88,9 +92,10 @@ RSpec.describe 'Counts API', type: :request do
end
it 'matches removed tags from the previous snapshot, not current post_tags' do
get '/counts/posts', params: { tag: tag.id, mode: 'details' }
get '/counts/posts', params: { tag: tag.id, latest: '0', mode: 'details' }
expect(response.parsed_body['count']).to eq(2)
expect(response.parsed_body['versions'].map { _1['version_no'] }).to eq([2, 1])
expect(response.parsed_body['versions'].first['tags'].sole['type']).to eq('removed')
expect(response.parsed_body['versions'].first['tags']).to be_empty
expect(response.parsed_body['versions'].last['tags'].sole['id']).to eq(tag.id)
end
end
+515
ファイルの表示
@@ -0,0 +1,515 @@
require 'rails_helper'
RSpec.describe 'Counts table API', type: :request do
include_context 'Ainu locale'
let!(:actor) { create(:user, :member, name: 'Counts actor') }
let!(:record) { create(:post) }
let!(:tag) { create(:tag, primary_name: 'alpha', category: :general) }
let!(:external) { create(:external_tag, name: 'beta') }
let(:point) { '2025-01-01T00:00:00Z' }
let(:base) do
{ resource: 'posts', definition: 'history', mode: 'details',
execution_at: point, timezone: 'UTC', locale: 'ja' }
end
before do
TagVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag:, version_no: 1, event_type: 'create', name: 'alpha',
category: 'general', aliases: '', parent_tag_ids: '',
names: { ja: { name: 'alpha', script_code: 'Jpan', auto_generated: false, aliases: [] } },
created_at: Time.utc(2023))
NicoTagVersion.create!(external_tag: external, version_no: 1,
event_id: create_event!(event_type: 'unknown').id,
event_type: 'create', linked_tag_ids: '', created_at: Time.utc(2023))
snapshots = [
{ tag_id: tag.id, version_no: 1, name: 'alpha', category: 'general', sections: [] },
{ external_tag_id: external.id }]
[1, 2, 3].each do |number|
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post: record, version_no: number,
event_type: number == 1 ? 'create' : 'update', url: record.url,
title: "snapshot #{ number }", tags: '', parent_post_ids: '',
tags_json: snapshots.first(number == 1 ? 2 : (number == 2 ? 1 : 0)),
created_by_user: number == 3 ? nil : actor,
created_at: Time.utc(2024, 1, 31, 15) + (number - 1).hours)
end
end
def query(extra = {})
get '/counts/query', params: base.merge(extra)
expect(response).to have_http_status(:ok)
response.parsed_body
end
def axis(field, value, type: 'string', grain: nil, root: 'conditions')
result = { root => '', "#{ root }.0.field" => field,
"#{ root }.0.value_type" => type }
result["#{ root }.0.operator"] = 'axis_eq' if root == 'conditions'
result["#{ root }.0.value"] = value unless type == 'null'
result["#{ root }.0.grain"] = grain if grain
result
end
def versions(result) = result.fetch('versions').map { _1.fetch('version_no') }
context 'nico link ID snapshots through the V1 field name' do
let(:base) do
super().merge(resource: 'nico_tags')
end
before do
NicoTagVersion.create!(external_tag: external, version_no: 2,
event_id: create_event!(event_type: 'unknown').id,
event_type: 'update', linked_tag_ids: tag.id.to_s,
created_at: Time.utc(2024, 1, 1))
NicoTagVersion.create!(external_tag: external, version_no: 3,
event_id: create_event!(event_type: 'unknown').id,
event_type: 'update', linked_tag_ids: '',
created_at: Time.utc(2024, 2, 1))
end
it 'returns ID values under linked_tags without exposing the physical column key' do
result = query
expect(versions(result)).to eq([1, 2, 3])
expect(result['versions'].map { _1.fetch('linked_tags') }).to eq(['', tag.id.to_s, ''])
result['versions'].each do |row|
expect(row).not_to have_key('linked_tag_ids')
expect(row.fetch('name')).to eq('nico:beta')
end
end
it 'preloads external names separately for aliased link-change details' do
statements = []
listener = lambda do |_name, _start, _finish, _id, payload|
statements << payload[:sql]
end
ActiveSupport::Notifications.subscribed(listener, 'sql.active_record') do
result = query(axis('changed_field', 'linked_tags'))
expect(versions(result)).to eq([2, 3])
expect(result.fetch('versions').map { _1.fetch('name') })
.to eq(['nico:beta', 'nico:beta'])
end
details = statements.find { _1.include?('AS counts_latest') }
expect(details).not_to be_nil
expect(details).not_to include('JOIN `external_tags`')
names = statements.select { _1.match?(/SELECT .* FROM `external_tags`/) }
expect(names.length).to eq(1)
end
it 'advertises the legacy field, label and default columns only' do
get '/counts/definitions/nico_tags', params: { locale: 'ja' }
expect(response).to have_http_status(:ok)
body = response.parsed_body
['fields', 'filter_fields', 'output_fields'].each do |collection|
keys = body.fetch(collection).map { _1.fetch('key') }
expect(keys).to include('linked_tags')
expect(keys).not_to include('linked_tag_ids')
end
field = body.fetch('output_fields').find { _1['key'] == 'linked_tags' }
expect(field.fetch('label')).to eq('counts:attributes.linked_tags')
body.fetch('axes').each_value do |keys|
expect(keys).to include('linked_tags')
expect(keys).not_to include('linked_tag_ids')
end
body.fetch('defaults').select { _1['mode'] == 'details' }.each do |entry|
settings = entry.fetch('params').to_h { [_1.fetch('key'), _1.fetch('value')] }
expect(settings.fetch('columns').split).to include('name', 'linked_tags')
expect(settings.fetch('columns').split).not_to include('linked_tag_ids')
end
end
it 'filters IDs and legacy drills and compares link changes through linked_tags' do
predicate = { conditions: '', 'conditions.0.field' => 'linked_tags',
'conditions.0.operator' => 'eq', 'conditions.0.value' => tag.id.to_s }
expect(versions(query(predicate))).to eq([2])
expect(versions(query(axis('linked_tags', tag.id.to_s, root: 'drill')))).to eq([2])
expect(versions(query(axis('changed_field', 'linked_tags')))).to eq([2, 3])
expect(versions(query(predicate.merge('conditions.0.value' => 'alpha')))).to be_empty
end
it 'sorts ID snapshots through the public field in both directions' do
expect(versions(query(sort: 'linked_tags:asc version_no:asc'))).to eq([1, 3, 2])
expect(versions(query(sort: 'linked_tags:desc version_no:asc'))).to eq([2, 1, 3])
end
it 'restores historical link IDs before applying a latest snapshot filter' do
predicate = { definition: 'latest', at: '2024-01-15T00:00:00Z',
conditions: '', 'conditions.0.field' => 'linked_tags',
'conditions.0.operator' => 'eq', 'conditions.0.value' => tag.id.to_s }
expect(versions(query(predicate))).to eq([2])
expect(versions(query(predicate.except(:at)))).to be_empty
end
it 'groups by linked_tags ID values and accepts the resulting drill condition' do
sign_in_as(create(:user, :admin))
result = query(mode: 'summary', rows: 'linked_tags', cols: '', sort: '')
expect(result.fetch('cells')).to match_array([
{ 'values' => [''], 'count' => 2 },
{ 'values' => [tag.id.to_s], 'count' => 1 }])
expect(versions(query(axis('linked_tags', tag.id.to_s)))).to eq([2])
end
it 'rejects the physical column as a public filter, axis, sort or change field' do
sign_in_as(create(:user, :admin))
[
axis('linked_tag_ids', tag.id.to_s),
axis('changed_field', 'linked_tag_ids'),
{ mode: 'summary', rows: 'linked_tag_ids', cols: '', sort: '' },
{ sort: 'linked_tag_ids:asc' }].each do |invalid|
get '/counts/query', params: base.merge(invalid)
expect(response).to have_http_status(:bad_request), invalid.inspect
end
end
end
it 'identifies a totals timeout without exposing SQL in the error response' do
allow_any_instance_of(CountsPivotTotals).to receive(:call)
.and_raise(CountsPivotTotals::TimedOut, 'private SQL')
get '/counts/query', params: base.merge(mode: 'summary', rows: 'created_at:month')
expect(response).to have_http_status(:service_unavailable)
expect(response.parsed_body).to eq('error' => 'counts_query_timeout',
'stage' => 'totals', 'timeout_seconds' => 30)
expect(response.body).not_to include('private SQL')
end
it 'applies the same optimizer safeguards to the paginated detail SELECT' do
statements = []
listener = lambda do |_name, _start, _finish, _id, payload|
statements << payload[:sql]
end
ActiveSupport::Notifications.subscribed(listener, 'sql.active_record') do
result = query(tags: 'alpha', limit: 1)
expect(result['count']).to eq(2)
expect(versions(result)).to eq([1])
end
details = statements.find { _1.include?('AS counts_latest') }
expect(details).not_to be_nil
expect(details).to include('MAX_EXECUTION_TIME(30000)',
"SET_VAR(optimizer_switch='semijoin=off')")
end
it 'separates latest and history and defaults to stable detail ordering and limit 100' do
history = query
expect(versions(history)).to eq([1, 2, 3])
expect(history).to include('limit' => 100, 'count' => 3, 'output_count' => 3,
'visible_output_count' => 3, 'excluded_count' => 0)
expect(history['versions'].map { _1['latest'] }).to eq([false, false, true])
expect(versions(query(definition: 'latest'))).to eq([3])
expect(versions(query(limit: 1, page: 2))).to eq([2])
expect(versions(query(sort: 'version_no:desc', limit: 1))).to eq([3])
end
it 'counts regular pivot rows separately from totals and from underlying records' do
total = query(mode: 'summary', rows: '', cols: '')
expect(total).to include('count' => 3, 'output_count' => 1, 'group_count' => 1)
expect(total['cells']).to eq([{ 'values' => [], 'count' => 3 }])
pivot = query(mode: 'summary', rows: 'event_type', cols: '', sort: '', limit: 1)
expect(pivot).to include('group_count' => 2, 'visible_output_count' => 2)
expect(pivot['row_headers'].count { _1['kind'] == 'data' }).to eq(1)
expect(pivot['row_headers']).to include(hash_including('kind' => 'grand_total'))
expect(query(mode: 'summary', rows: '', cols: 'event_type', sort: '')['output_count']).to eq(1)
end
it 'preserves timezone, NULL and legacy drill semantics and permits condition removal' do
month = axis('created_at', '2024-02', grain: 'month')
expect(versions(query(month.merge(timezone: 'Asia/Tokyo')))).to eq([1, 2, 3])
expect(versions(query(month))).to eq([])
old = axis('created_at', '2024-02', grain: 'month', root: 'drill')
expect(versions(query(old.merge(timezone: 'Asia/Tokyo')))).to eq([1, 2, 3])
expect(versions(query(old.merge(month).merge(timezone: 'Asia/Tokyo')))).to eq([1, 2, 3])
expect(versions(query(axis('user', nil, type: 'null')))).to eq([3])
expect(versions(query(axis('changed_field', 'title')))).to eq([2, 3])
expect(versions(query)).to eq([1, 2, 3])
end
it 'evaluates latest before filtering the version set' do
predicate = { conditions: '', 'conditions.0.field' => 'latest',
'conditions.0.operator' => 'eq', 'conditions.0.value' => 'false' }
expect(versions(query(predicate))).to eq([1, 2])
expect(versions(query(predicate.merge('conditions.0.value' => 'true')))).to eq([3])
end
it 'distinguishes the snapshot cutoff from the current latest flag and condition' do
past = { definition: 'latest', at: '2024-01-31T16:00:00Z' }
result = query(past)
expect(versions(result)).to eq([2])
expect(result['versions'].first['latest']).to be(false)
predicate = { conditions: '', 'conditions.0.field' => 'latest',
'conditions.0.operator' => 'eq', 'conditions.0.value' => 'false' }
expect(versions(query(past.merge(predicate)))).to eq([2])
expect(versions(query(past.merge(predicate).merge('conditions.0.value' => 'true'))))
.to eq([])
unchanged = query(definition: 'latest', at: '2024-02-01T00:00:00Z')
expect(versions(unchanged)).to eq([3])
expect(unchanged['versions'].first['latest']).to be(true)
expect(versions(query(sort: 'latest:desc resource_id:asc version_no:asc'))).to eq([3, 1, 2])
end
it 'checks the current latest flag beyond the execution timestamp' do
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post: record, version_no: 4, event_type: 'update',
url: record.url, title: 'after execution', tags: '', tags_json: [],
parent_post_ids: '', created_at: Time.utc(2025, 2, 1))
result = query
expect(versions(result)).to eq([1, 2, 3])
expect(result['versions'].map { _1['latest'] }).to eq([false, false, false])
snapshot = query(definition: 'latest')
expect(versions(snapshot)).to eq([3])
expect(snapshot['versions'].first['latest']).to be(false)
end
it 'keeps ANY, ALL and NONE distinct without multiplying parent records' do
values = { conditions: '', 'conditions.0.field' => 'tags',
'conditions.0.values' => '1', condition_values: '',
'condition_values.0.condition' => '0',
'condition_values.0.value_type' => 'string',
'condition_values.0.value' => "internal:#{ tag.id }",
'condition_values.1.condition' => '0',
'condition_values.1.value_type' => 'string',
'condition_values.1.value' => "external:#{ external.id }" }
{ 'any' => [1, 2], 'all' => [1], 'none' => [3] }.each do |operator, expected|
expect(versions(query(values.merge('conditions.0.operator' => operator)))).to eq(expected)
end
result = query(axis('tags', "internal:#{ tag.id }"))
expect(result['versions'].first['tags'].length).to eq(2)
end
it 'distinguishes IN, NOT IN, an empty selection and an unspecified selection' do
predicate = { conditions: '', 'conditions.0.field' => 'version_no',
'conditions.0.operator' => 'in', 'conditions.0.values' => '1',
condition_values: '', 'condition_values.0.condition' => '0',
'condition_values.0.value_type' => 'number',
'condition_values.0.value' => '1' }
expect(versions(query(predicate))).to eq([1])
expect(versions(query(predicate.merge('conditions.0.operator' => 'not_in')))).to eq([2, 3])
empty = predicate.reject { |key, _value| key.to_s.start_with?('condition_values.') }
expect(versions(query(empty))).to be_empty
expect(versions(query(empty.except('conditions.0.values')))).to eq([1, 2, 3])
end
it 'resolves current tag names for AND, OR and negative searches' do
expect(versions(query(tags: 'alpha'))).to eq([1, 2])
expect(versions(query(tags: 'alpha nico:beta'))).to eq([1])
expect(versions(query(tags: 'nico:beta alpha'))).to eq([1])
expect(versions(query(tags: 'alpha not:nico:beta'))).to eq([2])
expect(versions(query(tags: 'alpha nico:beta', match: 'any'))).to eq([1, 2])
expect(versions(query(tags: 'not:alpha'))).to eq([3])
end
context 'when current tag names differ from recorded snapshot names' do
before do
tag.tag_names.find_by!(language_code: 'ja', primary_flg: true)
.update!(name: 'renamed-alpha')
create(:tag_name, :alias, tag:, name: 'alpha-alias')
TagVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag:, version_no: 2, event_type: 'update', name: 'renamed-alpha',
category: 'general', aliases: 'alpha-alias', parent_tag_ids: '',
names: { ja: { name: 'renamed-alpha', script_code: 'Jpan', auto_generated: false,
aliases: [{ name: 'alpha-alias', script_code: 'Jpan' }] } },
created_at: Time.utc(2024, 2, 1))
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post: record, version_no: 4, event_type: 'update',
url: record.url, title: 'after rename', tags: '', parent_post_ids: '',
tags_json: [{ tag_id: tag.id, version_no: 2, name: 'renamed-alpha',
category: 'general', sections: [] }],
created_by_user: actor, created_at: Time.utc(2024, 2, 2))
end
it 'resolves current names and aliases to stable IDs in both definitions' do
{ 'history' => [1, 2, 4], 'latest' => [4] }.each do |definition, expected|
['renamed-alpha', 'alpha-alias'].each do |name|
expect(versions(query(tags: name, definition:))).to eq(expected)
end
end
result = query(tags: 'renamed-alpha')
expect(result['versions'].first['tags'].first['name']).to eq('alpha')
expect(versions(query(tags: 'renamed-alpha', definition: 'latest',
at: '2024-01-31T16:00:00Z'))).to eq([2])
expect(versions(query(tags: 'not:alpha-alias'))).to eq([3])
expect(versions(query(tags: 'renamed-alpha nico:beta'))).to eq([1])
expect(versions(query(tags: 'renamed-alpha nico:beta', match: 'any'))).to eq([1, 2, 4])
end
it 'does not match a historical name reassigned to another tag ID' do
create(:tag, primary_name: 'alpha')
expect(versions(query(tags: 'alpha'))).to eq([])
expect(versions(query(tags: 'unregistered-name'))).to eq([])
expect(versions(query(tags: 'not:unregistered-name'))).to eq([1, 2, 3, 4])
expect(versions(query(tags: 'renamed-alpha'))).to eq([1, 2, 4])
end
it 'resolves immutable external names across historical snapshots' do
expect(versions(query(tags: 'nico:beta'))).to eq([1])
expect(versions(query(tags: 'nico:renamed-beta'))).to eq([])
expect(versions(query(tags: 'not:nico:beta'))).to eq([2, 3, 4])
end
end
it 'rejects malformed structured conditions and axes without accepting arbitrary SQL' do
[
{ rows: '[{"field":"user"}]' }, { rows: 'user user' },
{ rows: 'user:month' }, { rows: 'password' }, { limit: 101 }, { limit: 0 },
{ page: '1x' }, { conditions: 'eyJmaWVsZCI6InVzZXIifQ==' },
{ conditions: '', 'conditions.1.field' => 'user' },
axis('password', '1'), axis('changed_field', 'title').merge(definition: 'latest'),
axis('user', 'yes', type: 'boolean'),
{ conditions: '', 'conditions.0.field' => 'title',
'conditions.0.operator' => 'eq', 'conditions.0.value' => 'x' * 8193 }].each do |invalid|
get '/counts/query', params: base.merge(invalid)
expect(response).to have_http_status(:bad_request), invalid.inspect
end
end
it 'rejects duplicate URL keys and JSON query payloads at the HTTP boundary' do
['resource=posts&tag_refs=', 'resource=posts&tag_refs.0.tag_id=1',
'resource=posts&limit=20&limit=50', 'resource=posts&headers=%5B%22Id.%22%5D',
'resource=posts&limit%5B%5D=20'].each do |query_string|
get "/counts/query?#{ query_string }"
expect(response).to have_http_status(:bad_request)
end
end
it 'reports column overflow separately from malformed input' do
101.times do |index|
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post: record, version_no: index + 4,
event_type: 'update', url: record.url, title: "column #{ index }",
tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024, 3))
end
get '/counts/query', params: base.merge(mode: 'summary', rows: '', cols: 'title', sort: '')
expect(response).to have_http_status(:unprocessable_entity)
expect(response.parsed_body).to eq('error' => 'too_many_columns', 'max_columns' => 100)
end
it 'defines all six resources with filterable output fields and shared detail defaults' do
locale
CountsResourceDefinition::DEFINITIONS.each_key do |resource|
get "/counts/definitions/#{ resource }", params: { locale: 'ain-Kana' }
expect(response).to have_http_status(:ok)
body = response.parsed_body
filters = body['filter_fields'].map { _1['key'] }
walk = lambda do |fields, prefix|
fields.each do |field|
key = [prefix, field['key']].compact.join('.')
expect(filters).to include(key)
walk.call(field.fetch('children', []), key)
end
end
walk.call(body['output_fields'], nil)
defaults = body['defaults'].select { _1['mode'] == 'details' }
expect(defaults.map { _1['definition'] }).to eq(['latest', 'history'])
expect(defaults.first['params']).to eq(defaults.last['params'])
settings = defaults.first['params'].to_h { [_1['key'], _1['value']] }
expect(settings['sort']).to eq('resource_id:asc version_no:asc')
if resource == 'posts'
expect(settings['children'].split).to eq(['tags.name_ain', 'parents.id'])
end
keys = body['name_columns'].map { _1['key'] }
languages = Language.order(:created_at, :code).pluck(:code)
expect(keys).to eq(languages.map { "name_#{ _1 }" } + languages.map { "aliases_#{ _1 }" })
end
end
[nil, :guest, :member, :admin].each do |role|
context "authority #{ role || 'anonymous' }" do
before do
sign_in_as(create(:user, role)) if role
# Equal timestamps deliberately require version ordering, not a date comparison.
['discard', 'restore', 'discard', 'restore'].each_with_index do |event, index|
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post: record, version_no: index + 4, event_type: event,
url: record.url, title: event == 'restore' ? 'restored' : 'private snapshot',
tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024, 3))
end
end
it 'keeps every earlier discard boundary private but counts denied records' do
result = query
expected = role == :admin ? (1..7).to_a : [7]
expect(versions(result)).to eq(expected)
expect(result).to include('output_count' => 7,
'visible_output_count' => expected.length, 'excluded_count' => 7 - expected.length)
expect(response.headers['Cache-Control']).to include('no-store')
expect(response.body).not_to include('private snapshot') unless role == :admin
end
it 'applies the same version boundary to the ordinary history endpoint and old values' do
get '/posts/versions', params: { post: record.id, locale: 'ja' }
expect(response).to have_http_status(:ok)
rows = response.parsed_body.fetch('versions')
expect(rows.map { _1['version_no'] }).to eq(role == :admin ? (1..7).to_a.reverse : [7])
unless role == :admin
expect(rows.sole.dig('title', 'prev')).to be_nil
expect(response.body).not_to include('private snapshot')
end
end
it 'does not resolve private condition names even when the result set is empty' do
TagVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag:, version_no: 2, event_type: 'discard', name: 'secret tag',
category: 'general', aliases: '', parent_tag_ids: '',
names: { ja: { name: 'secret tag', script_code: 'Jpan', auto_generated: false, aliases: [] } },
created_at: Time.utc(2024, 2))
result = query(axis('tags', "internal:#{ tag.id }").merge(definition: 'latest'))
expect(result['versions']).to be_empty
expected = role == :admin ? { "internal:#{ tag.id }" => 'secret tag' } : {}
expect(result['condition_labels']['tags']).to eq(expected)
get '/counts/definitions/posts/candidates', params: {
field: 'tags', query: 'secret', execution_at: point, locale: 'ja' }
expect(response).to have_http_status(:ok)
expect(response.parsed_body['rows']).to be_empty
end
end
end
it 'includes private history in the public timeline without an exclusion count' do
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post: record, version_no: 4, event_type: 'discard', url: record.url,
title: 'hidden title', tags: '', tags_json: [], parent_post_ids: '',
created_at: Time.utc(2024, 3))
result = query(mode: 'summary', rows: 'created_at:month', cols: 'event_type')
expect(result['excluded_count']).to be_nil
expect(result['count']).to eq(4)
expect(result['cells'].sum { _1['count'] }).to eq(4)
expect(response.body).not_to include('hidden title')
masked = query(mode: 'summary', rows: 'resource_id', cols: '')
expect(masked['cells'].map { _1['values'] }).to eq([[record.id]])
expect(masked['cells'].first['count']).to eq(4)
expect(query['versions']).to be_empty
end
it 'allows public identity predicates without exposing private snapshot fields' do
PostVersion.create!(post: record, version_no: 4, event_type: 'discard',
url: record.url, title: 'hidden title', tags: '', tags_json: [],
parent_post_ids: '', created_at: Time.utc(2024, 3))
get '/counts/query', params: base.merge(mode: 'summary',
resource_id: record.id.to_s, rows: 'resource_id', cols: 'title')
expect(response).to have_http_status(:ok)
expect(json['count']).to eq(4)
expect(json['cells'].sum { _1['count'] }).to eq(4)
expect(json['cells'].map { _1['values'].last })
.to include(CountsAxes::MASKED_VALUE)
expect(response.body).not_to include('hidden title')
end
it 'reports the combined axis limit without replacing it with a generic 400' do
get '/counts/query', params: base.merge(mode: 'summary', sort: '',
rows: 'resource_id version_no event_type user created_at:month',
cols: 'title url thumbnail_base video_ms')
expect(response).to have_http_status(:bad_request)
expect(response.parsed_body).to eq('error' => 'too_many_axes', 'max_axes' => 8)
end
end
+40 -1
ファイルの表示
@@ -314,6 +314,7 @@ RSpec.describe 'Materials API', type: :request do
.and change(Tag, :count).by(1)
.and change(TagName, :count).by(1)
.and change(MaterialVersion, :count).by(1)
.and change { event_rows.length }.by(1)
expect(response).to have_http_status(:created)
@@ -325,6 +326,12 @@ RSpec.describe 'Materials API', type: :request do
expect(material.file.attached?).to be(true)
expect(material.version_no).to eq(1)
expect(material.material_versions.first.event_type).to eq('create')
event = event_rows.last
expect(event).to include(
'event_type' => 'materials.create',
'created_by_user_id' => member_user.id)
expect(material.material_versions.first.event_id).to eq(event.fetch('id'))
expect(material.tag.tag_versions.pluck(:event_id)).to eq([event.fetch('id')])
expect(material.material_export_items.first.export_path).to eq('伊地知ニジカ/created.png')
expect(material.material_versions.first.export_paths_json).to eq(
'legacy_drive' => '伊地知ニジカ/created.png'
@@ -509,6 +516,7 @@ RSpec.describe 'Materials API', type: :request do
export_paths: { legacy_drive: '伊地知ニジカ/updated.jpg' }
}
end.to change(MaterialVersion, :count).by(2)
.and change { event_rows.length }.by(1)
expect(response).to have_http_status(:ok)
@@ -517,6 +525,12 @@ RSpec.describe 'Materials API', type: :request do
expect(material.tag.category).to eq('material')
expect(material.url).to eq('https://example.com/updated-source')
expect(material.updated_by_user).to eq(member_user)
event = event_rows.last
expect(event).to include(
'event_type' => 'materials.update',
'created_by_user_id' => member_user.id)
expect(material.material_versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
expect(material.tag.tag_versions.pluck(:event_id)).to eq([event.fetch('id')])
expect(material.file.attached?).to be(true)
expect(material.file.blob.id).not_to eq(old_blob_id)
expect(ActiveStorage::Blob.where(id: old_blob_id).exists?).to be(true)
@@ -632,6 +646,7 @@ RSpec.describe 'Materials API', type: :request do
MaterialVersionRecorder.record!(material:, event_type: :create,
created_by_user: member_user)
TagVersioning.ensure_snapshot!(tag, created_by_user: member_user)
events = event_rows
expect do
put "/materials/#{ material.id }", params: {
@@ -643,6 +658,7 @@ RSpec.describe 'Materials API', type: :request do
expect(response).to have_http_status(:ok)
expect(material.reload.version_no).to eq(1)
expect(event_rows).to eq(events)
end
it 'records update version when only export_path changes' do
@@ -807,6 +823,12 @@ RSpec.describe 'Materials API', type: :request do
it 'creates source suppression as member' do
sign_in_as(member_user)
materials = ['a', 'b'].map do |name|
Material.create!(
source_kind: 'google_drive_file', source_file_id: "blocked-#{ name }",
source_path: "伊地知ニジカ/危険素材/#{ name }.png",
url: "https://example.com/blocked-#{ name }", created_by_user: member_user)
end
expect do
post '/materials/suppressions', params: {
@@ -815,6 +837,7 @@ RSpec.describe 'Materials API', type: :request do
reason: 'copyright_high_risk'
}
end.to change(MaterialSyncSuppression, :count).by(1)
.and change { event_rows.length }.by(1)
expect(response).to have_http_status(:created)
expect(json).to include(
@@ -822,6 +845,14 @@ RSpec.describe 'Materials API', type: :request do
'drive_path' => '伊地知ニジカ/危険素材',
'normalized_source_key' => 'google_drive_path_prefix:伊地知ニジカ/危険素材'
)
event = event_rows.last
expect(event).to include(
'event_type' => 'materials.block', 'created_by_user_id' => member_user.id)
expect(materials.map { _1.reload.discarded? }).to eq([true, true])
versions = MaterialVersion.where(material_id: materials.map(&:id))
expect(versions.where(event_type: 'discard').pluck(:material_id))
.to match_array(materials.map(&:id))
expect(versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
end
end
@@ -860,11 +891,19 @@ RSpec.describe 'Materials API', type: :request do
end
it 'discards the material and returns 204' do
delete "/materials/#{ material.id }"
expect { delete "/materials/#{ material.id }" }
.to change { event_rows.length }.by(1)
expect(response).to have_http_status(:no_content)
expect(Material.find_by(id: material.id)).to be_nil
expect(Material.with_discarded.find(material.id)).to be_discarded
event = event_rows.last
expect(event).to include(
'event_type' => 'materials.destroy',
'created_by_user_id' => member_user.id)
versions = MaterialVersion.where(material_id: material.id)
expect(versions.order(:version_no).last.event_type).to eq('discard')
expect(versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
end
end
end
+14 -14
ファイルの表示
@@ -38,12 +38,9 @@ RSpec.describe 'NicoTags', type: :request do
end
it 'filters by nico tag name, linked tag name, and link status' do
linked = create(:external_tag)
linked.update!(name: 'search_linked')
unlinked = create(:external_tag)
unlinked.update!(name: 'search_unlinked')
other = create(:external_tag)
other.update!(name: 'other')
linked = create(:external_tag, name: 'search_linked')
unlinked = create(:external_tag, name: 'search_unlinked')
other = create(:external_tag, name: 'other')
destination = create(:tag, category: :general)
primary_tag_name_for(destination, 'ja').update!(name: 'destination_search')
NicoTagRelation.create!(nico_tag: linked, tag: destination)
@@ -80,11 +77,9 @@ RSpec.describe 'NicoTags', type: :request do
end
it 'sorts by name and timestamps' do
older = create(:external_tag)
older.update!(name: 'ordered_a')
older = create(:external_tag, name: 'ordered_a')
older.update_columns(created_at: 2.days.ago)
newer = create(:external_tag)
newer.update!(name: 'ordered_b')
newer = create(:external_tag, name: 'ordered_b')
newer.update_columns(created_at: 1.day.ago)
older_post_tag =
PostExternalTag.create!(post: create(:post), external_tag: older)
@@ -177,6 +172,7 @@ RSpec.describe 'NicoTags', type: :request do
}
}.to change(TagVersion, :count).by(2)
.and change(NicoTagVersion, :count).by(1)
.and change { event_rows.length }.by(1)
expect(response).to have_http_status(:ok)
@@ -186,13 +182,17 @@ RSpec.describe 'NicoTags', type: :request do
linked_versions = TagVersion.where(tag: [linked_a, linked_b]).order(:tag_id)
expect(linked_versions.map(&:event_type)).to eq(['create', 'create'])
expect(linked_versions.map(&:created_by_user_id)).to all(eq(admin.id))
event = event_rows.last
expect(event).to include(
'event_type' => 'external_tags.update', 'created_by_user_id' => admin.id)
expect(linked_versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
versions = nico_tag.reload.nico_tag_versions.order(:version_no)
expect(versions.map(&:event_type)).to eq(['create', 'update'])
expect(versions.last.linked_tags.split).to match_array([
'nico_linked_a',
'nico_linked_b'])
expect(versions.last.linked_tag_ids.split.map(&:to_i))
.to match_array([linked_a.id, linked_b.id])
expect(versions.last.created_by_user_id).to eq(admin.id)
expect(versions.last.event_id).to eq(event.fetch('id'))
end
it 'clears existing links and records the empty mapping for a member' do
@@ -210,7 +210,7 @@ RSpec.describe 'NicoTags', type: :request do
expect(json).to eq([])
expect(nico_tag.reload.linked_tags).to be_empty
expect(nico_tag.nico_tag_versions.order(:version_no).last)
.to have_attributes(linked_tags: '', created_by_user: member)
.to have_attributes(linked_tag_ids: '', created_by_user: member)
end
it 'returns the tags field error when a nico tag is specified directly' do
+130
ファイルの表示
@@ -0,0 +1,130 @@
require 'rails_helper'
RSpec.describe 'Post history privacy', type: :request do
let(:origin) { Time.utc(2020, 1, 1) }
let!(:post_record) do
create(:post, title: 'current', created_at: origin, updated_at: origin)
end
let!(:other_post) do
create(:post, created_at: origin, updated_at: origin + 3.days + 1.hour)
end
let!(:tag) { create(:tag, primary_name: 'private-history-tag') }
let!(:external) { create(:external_tag, name: 'external-history-tag') }
before do
['create', 'discard'].each_with_index do |event, index|
TagVersion.create!(tag:, version_no: index + 1, event_type: event,
name: 'private-history-tag', category: 'general', aliases: '', parent_tag_ids: '',
names: { ja: { name: 'private-history-tag', script_code: 'Jpan',
auto_generated: false, aliases: [] } },
created_at: origin + index.hours)
end
NicoTagVersion.create!(external_tag: external, version_no: 1, event_type: 'create',
event_id: create_event!(event_type: 'unknown').id,
linked_tag_ids: tag.id.to_s,
created_at: origin)
['create', 'discard', 'restore', 'update'].each_with_index do |event, index|
snapshots =
if index < 2
[{ tag_id: tag.id, version_no: 1, name: 'private-history-tag', sections: [] }]
elsif index == 3
[{ external_tag_id: external.id }]
else
[]
end
PostVersion.create!(post: post_record, version_no: index + 1, event_type: event,
title: "version #{ index + 1 }", url: post_record.url, tags: '',
tags_json: snapshots, parent_post_ids: '', created_at: origin + (index + 1).days)
end
end
it 'evaluates each reference condition against the restored snapshot alone' do
HistoryAccess.set(user: nil) do
restored = PostVersion.where(post_id: post_record.id, version_no: 3)
HistoryAccess.reference_sql('post_versions', 'post_versions').each do |condition|
expect(restored.where(Arel.sql(condition)).pluck(:version_no)).to eq([3])
end
expect(HistoryAccess.visible(PostVersion.where(post_id: post_record.id))
.pluck(:version_no)).to eq([3])
end
end
it 'keeps every version through a later discard private after another restore' do
['discard', 'restore'].each_with_index do |event, index|
PostVersion.create!(post: post_record, version_no: index + 5, event_type: event,
title: 'later snapshot', url: post_record.url, tags: '', tags_json: [],
parent_post_ids: '', created_at: origin + (index + 5).days)
end
get '/posts/versions', params: { post: post_record.id }
expect(response).to have_http_status(:ok)
expect(response.parsed_body.fetch('versions').pluck('version_no')).to eq([6])
expect(response.parsed_body.fetch('count')).to eq(1)
expect(response.parsed_body.fetch('versions').first.fetch('title').fetch('prev')).to be_nil
end
['anonymous', 'member', 'admin'].each do |role|
context "as #{ role }" do
before do
sign_in_as(create(:user, role.to_sym)) unless role == 'anonymous'
end
it 'uses only authorised history timestamps for display' do
get '/posts'
expect(response).to have_http_status(:ok)
row = response.parsed_body.fetch('posts').find { _1.fetch('id') == post_record.id }
expected = origin + (role == 'admin' ? 4 : 3).days
expect(Time.zone.parse(row.fetch('updated_at'))).to eq(expected)
end
it 'uses the same visibility for date bounds and count' do
get '/posts', params: { updated_from: (origin + 4.days).iso8601,
updated_to: (origin + 4.days).iso8601 }
expect(response).to have_http_status(:ok)
ids = response.parsed_body.fetch('posts').pluck('id')
expect(ids).to eq(role == 'admin' ? [post_record.id] : [])
expect(response.parsed_body.fetch('count')).to eq(ids.length)
end
it 'sorts using authorised timestamps in both directions' do
['asc', 'desc'].each do |direction|
get '/posts', params: { order: "updated_at:#{ direction }" }
expect(response).to have_http_status(:ok)
expected = role == 'admin' ? [other_post.id, post_record.id] :
[post_record.id, other_post.id]
expected.reverse! if direction == 'desc'
expect(response.parsed_body.fetch('posts').pluck('id')).to eq(expected)
end
end
it 'applies authorised timestamp ordering before pagination' do
get '/posts', params: { order: 'updated_at:desc', limit: 1 }
expect(response).to have_http_status(:ok)
expected = role == 'admin' ? post_record.id : other_post.id
expect(response.parsed_body.fetch('posts').pluck('id')).to eq([expected])
expect(response.parsed_body.fetch('count')).to eq(2)
end
it 'counts only visible versions and masks a private previous version' do
get '/posts/versions', params: { post: post_record.id }
expect(response).to have_http_status(:ok)
body = response.parsed_body
expected = role == 'admin' ? [4, 3, 2, 1] : [3]
expect(body.fetch('count')).to eq(expected.length)
expect(body.fetch('versions').pluck('version_no')).to eq(expected)
restored = body.fetch('versions').find { _1.fetch('version_no') == 3 }
expect(restored.fetch('title').fetch('prev')).to eq(role == 'admin' ? 'version 2' : nil)
end
it 'uses only an authorised previous version for tag filtering' do
get '/posts/versions', params: { post: post_record.id, tag: tag.id }
expect(response).to have_http_status(:ok)
expected = role == 'admin' ? [3, 2, 1] : []
expect(response.parsed_body.fetch('versions').pluck('version_no')).to eq(expected)
expect(response.parsed_body.fetch('count')).to eq(expected.length)
end
end
end
end
+26 -11
ファイルの表示
@@ -11,34 +11,49 @@ RSpec.describe 'Post filter identity across locales', type: :request do
let!(:other_post) { create(:post) }
before do
# Legacy data can predate the current prohibition on internal nico: names.
primary_tag_name_for(tag, 'ja').update_columns(name: 'nico:shared')
primary_tag_name_for(tag, 'ain').update!(name: 'シロシ', auto_generated: false)
create(:tag_name, :alias, tag:, name: 'union_alias', language_code: 'ja')
PostTag.create!(post: internal_post, tag:)
PostExternalTag.create!(post: external_post, external_tag: external)
end
['nico:shared', 'union_alias', 'not:nico:shared'].each do |literal|
literals = ['legacy_union_source', 'union_alias', 'not:legacy_union_source', 'nico:shared']
literals.each do |literal|
it "preserves the complete match set for #{ literal }" do
get '/posts', params: { locale: 'ja', tags: literal }
expect(response).to have_http_status(:ok)
original_ids = json.fetch('posts').map { _1.fetch('id') }
expected = literal.start_with?('not:') ? [other_post.id] :
[internal_post.id, external_post.id]
expected = if literal.start_with?('not:')
[external_post.id, other_post.id]
elsif literal == 'nico:shared'
[external_post.id]
else
[internal_post.id]
end
expect(original_ids).to match_array(expected)
get '/posts/tag_filters', params: {
locale: 'ja', target_locale: 'ain-Kana', tags: literal }
expect(response).to have_http_status(:ok)
references = json.fetch('references')
expect(references.first).to include('tag_id' => tag.id, 'external_tag_id' => external.id)
get '/posts', params: {
locale: 'ain-Kana', tags: references.map { _1.fetch('name') }.join(' '),
tag_refs: references.to_json }
translated = json.fetch('tags')
expect(translated).to eq(literal == 'nico:shared' ? literal :
(literal.start_with?('not:') ? 'not:シロシ' : 'シロシ'))
get '/posts', params: { locale: 'ain-Kana', tags: translated }
expect(response).to have_http_status(:ok)
expect(json.fetch('posts').map { _1.fetch('id') }).to match_array(original_ids)
end
end
it 'rejects a locale conversion that cannot preserve a legacy union' do
primary_tag_name_for(tag, 'ja').update_columns(name: 'nico:shared')
get '/posts/tag_filters', params: {
locale: 'ja', target_locale: 'ain-Kana', tags: 'nico:shared' }
expect(response).to have_http_status(:unprocessable_entity)
end
it 'rejects removed reference parameters instead of overriding the name' do
get '/posts', params: { locale: 'ja', tags: 'legacy_union_source',
tag_refs: '', 'tag_refs.0.tag_id' => external.id.to_s }
expect(response).to have_http_status(:bad_request)
end
end
+80 -16
ファイルの表示
@@ -50,6 +50,8 @@ RSpec.describe 'Posts API', type: :request do
post:,
version_no: 1,
event_type: 'create',
event_id: create_event!(
event_type: 'posts.create', created_by_user: post.uploaded_user).id,
title: post.title,
url: post.url,
thumbnail_base: post.thumbnail_base,
@@ -629,6 +631,37 @@ RSpec.describe 'Posts API', type: :request do
create(:post_tag, post: plain_post, tag:, created_at: t3)
end
it 'displays history time after pagination without joining history to posts or count' do
queries = []
subscriber = ->(event) { queries << event.payload.fetch(:sql) }
ActiveSupport::Notifications.subscribed(subscriber, 'sql.active_record') do
get '/posts', params: { url: '/version-time/', order: 'url:asc', limit: 1 }
end
expect(response).to have_http_status(:ok)
expect(json.fetch('posts').pluck('id')).to eq([history_post.id])
expect(Time.zone.parse(json.fetch('posts').first.fetch('updated_at'))).to eq(t2)
expect(json.fetch('count')).to eq(3)
post_queries = queries.select { _1.include?('FROM `posts`') }
expect(post_queries).not_to be_empty
expect(post_queries.grep(/post_versions/)).to be_empty
end
it 'omits the history aggregate from count when only sorting by update time' do
queries = []
subscriber = ->(event) { queries << event.payload.fetch(:sql) }
ActiveSupport::Notifications.subscribed(subscriber, 'sql.active_record') do
get '/posts', params: { url: '/version-time/', order: 'updated_at:desc', limit: 1 }
end
expect(response).to have_http_status(:ok)
expect(json.fetch('posts').pluck('id')).to eq([newer_post.id])
expect(json.fetch('count')).to eq(3)
count_queries = queries.select { _1.include?('FROM `posts`') && _1.match?(/COUNT\(/i) }
expect(count_queries).not_to be_empty
expect(count_queries.grep(/post_versions/)).to be_empty
end
['asc', 'desc'].each do |direction|
it "sorts by the later of post update and latest version time (#{ direction })" do
get '/posts', params: { url: '/version-time/', order: "updated_at:#{ direction }" }
@@ -658,6 +691,16 @@ RSpec.describe 'Posts API', type: :request do
expect(json.fetch('posts').map { |item| item.fetch('id') }).to eq([history_post.id])
expect(json.fetch('count')).to eq(1)
end
it 'counts every timestamp match before pagination' do
get '/posts', params: { url: '/version-time/', order: 'updated_at:asc',
updated_from: t1.iso8601, updated_to: t3.iso8601,
limit: 1, page: 2 }
expect(response).to have_http_status(:ok)
expect(json.fetch('posts').pluck('id')).to eq([history_post.id])
expect(json.fetch('count')).to eq(3)
end
end
context 'when original_created_from/original_created_to are provided' do
@@ -1027,7 +1070,7 @@ RSpec.describe 'Posts API', type: :request do
it 'dry-runs without persisting posts or new tags' do
sign_in_as(member)
counts = [Post.count, Tag.count, TagName.count]
counts = [Post.count, Tag.count, TagName.count, event_rows.length]
post '/posts?dry=1', params: post_write_params(
title: 'dry-run post',
@@ -1043,7 +1086,7 @@ RSpec.describe 'Posts API', type: :request do
[{ 'name' => 'new_dry_run_tag',
'category' => 'character',
'section_literals' => [] }])
expect([Post.count, Tag.count, TagName.count]).to eq(counts)
expect([Post.count, Tag.count, TagName.count, event_rows.length]).to eq(counts)
end
it 'dry-runs an existing URL without validating its upload' do
@@ -2218,20 +2261,20 @@ RSpec.describe 'Posts API', type: :request do
def create_post_version! post, version_no:, event_type:, created_by_user:, created_at:
PostVersion.create!(
post:,
version_no:,
event_type:,
title: post.title,
url: post.url,
thumbnail_base: post.thumbnail_base,
tags: snapshot_tags(post),
tags_json: post.snapshot_tags_json,
parent_post_ids: post.snapshot_parent_post_ids.join(' '),
original_created_from: post.original_created_from,
original_created_before: post.original_created_before,
created_at:,
created_by_user:
)
event_id: create_event!(event_type: 'unknown').id,
post:,
version_no:,
event_type:,
title: post.title,
url: post.url,
thumbnail_base: post.thumbnail_base,
tags: snapshot_tags(post),
tags_json: post.snapshot_tags_json,
parent_post_ids: post.snapshot_parent_post_ids.join(' '),
original_created_from: post.original_created_from,
original_created_before: post.original_created_before,
created_at:,
created_by_user:)
end
let!(:v1) do
@@ -2559,6 +2602,7 @@ RSpec.describe 'Posts API', type: :request do
tags: 'spec_tag',
thumbnail: dummy_upload)
end.to change(PostVersion, :count).by(1)
.and change { event_rows.length }.by(1)
expect(response).to have_http_status(:created)
@@ -2570,6 +2614,13 @@ RSpec.describe 'Posts API', type: :request do
expect(version.url).to eq('https://example.com/versioned-post')
expect(version.created_by_user_id).to eq(member.id)
expect(version.tags).to eq(snapshot_tags(created_post))
event = event_rows.last
expect(event).to include(
'event_type' => 'posts.create', 'created_by_user_id' => member.id)
expect(version.event_id).to eq(event.fetch('id'))
tag_versions = TagVersion.where(tag_id: created_post.tags.select(:id))
expect(tag_versions).not_to be_empty
expect(tag_versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
end
it 'creates next version on PUT /posts/:id when snapshot changes' do
@@ -2585,6 +2636,7 @@ RSpec.describe 'Posts API', type: :request do
title: 'updated title',
tags: 'spec_tag_2')
end.to change(PostVersion, :count).by(1)
.and change { event_rows.length }.by(1)
expect(response).to have_http_status(:ok)
@@ -2594,6 +2646,12 @@ RSpec.describe 'Posts API', type: :request do
expect(version.title).to eq('updated title')
expect(version.created_by_user_id).to eq(member.id)
expect(version.tags).to eq(snapshot_tags(post_record.reload))
event = event_rows.last
expect(event).to include(
'event_type' => 'posts.update', 'created_by_user_id' => member.id)
expect(version.event_id).to eq(event.fetch('id'))
tag_version = TagVersion.find_by!(tag_id: tag_name2.reload.tag_id, version_no: 1)
expect(tag_version.event_id).to eq(event.fetch('id'))
end
it 'does not create a new version on PUT /posts/:id when snapshot is unchanged' do
@@ -2602,6 +2660,7 @@ RSpec.describe 'Posts API', type: :request do
PostTag.create!(post: post_record, tag: Tag.no_deerjikist)
base_version = create_post_version_for!(post_record.reload)
events = event_rows
expect {
put "/posts/#{post_record.id}", params: post_write_params(
base_version_no: base_version.version_no,
@@ -2615,10 +2674,12 @@ RSpec.describe 'Posts API', type: :request do
expect(version.version_no).to eq(1)
expect(version.event_type).to eq('create')
expect(version.tags).to eq(snapshot_tags(post_record))
expect(event_rows).to eq(events)
end
it 'does not create a version when POST /posts is invalid' do
sign_in_as(member)
events = event_rows
expect do
post '/posts', params: post_write_params(
@@ -2629,11 +2690,13 @@ RSpec.describe 'Posts API', type: :request do
end.not_to change(PostVersion, :count)
expect(response).to have_http_status(:unprocessable_entity)
expect(event_rows).to eq(events)
end
it 'does not create a version when PUT /posts/:id is invalid' do
sign_in_as(member)
base_version = create_post_version_for!(post_record)
events = event_rows
expect do
put "/posts/#{post_record.id}", params: post_write_params(
@@ -2645,6 +2708,7 @@ RSpec.describe 'Posts API', type: :request do
end.not_to change(PostVersion, :count)
expect(response).to have_http_status(:unprocessable_entity)
expect(event_rows).to eq(events)
end
it 'rejects second-bearing original created timestamps on POST /posts' do
+3 -1
ファイルの表示
@@ -119,7 +119,9 @@ RSpec.describe 'Sitemap entries API', type: :request do
# Legacy empty bodies can exist even though the write service rejects them.
empty = TagDescription.create!(tag: empty_tag, language_code: 'ain',
body: '', version_no: 1)
TagDescriptionVersion.create!(tag_id: empty_tag.id, language_code: 'ain',
TagDescriptionVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag_id: empty_tag.id, language_code: 'ain',
body: '', version_no: 1, event_type: 'create',
created_by_user_id: user.id, created_at: Time.current)
+10
ファイルの表示
@@ -53,8 +53,13 @@ RSpec.describe "TagChildren", type: :request do
expect { do_request }
.to change { parent.reload.children.ids.include?(child.id) }
.from(false).to(true)
.and change { event_rows.length }.by(1)
expect(response).to have_http_status(:no_content)
event = event_rows.last
expect(event).to include(
'event_type' => 'tags.update', 'created_by_user_id' => admin.id)
expect(child.tag_versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
end
it 'returns 422 and does not create relation when the new link makes a cycle' do
@@ -132,6 +137,7 @@ RSpec.describe "TagChildren", type: :request do
expect {
do_request
}.to change(TagVersion, :count).by(2)
.and change { event_rows.length }.by(1)
expect(response).to have_http_status(:no_content)
@@ -140,6 +146,10 @@ RSpec.describe "TagChildren", type: :request do
expect(versions.first.parent_tag_ids.split).to include(parent.id.to_s)
expect(versions.second.parent_tag_ids).to eq('')
expect(versions.second.created_by_user_id).to eq(admin.id)
event = event_rows.last
expect(event).to include(
'event_type' => 'tags.update', 'created_by_user_id' => admin.id)
expect(versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
end
end
+5 -3
ファイルの表示
@@ -8,6 +8,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do
let!(:versions) do
['A', 'A', 'new-version'].each_with_index.map do |body, index|
TagDescriptionVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag_id: tag.id, language_code: 'ja', version_no: index + 1,
event_type: index.zero? ? 'create' : 'update', body:,
created_by_user_id: user.id, created_at: Time.current)
@@ -55,7 +56,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do
end
it 'does not fall back to a revision absent from new history' do
versions.first.destroy!
TagDescriptionVersion.where(id: versions.first.id).delete_all
without_legacy_reads do
get "/wiki/#{ description.id }", params: { version: versions.first.id }
end
@@ -63,7 +64,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do
end
it 'does not fall back when the new current revision is missing' do
versions.last.destroy!
TagDescriptionVersion.where(id: versions.last.id).delete_all
without_legacy_reads { get "/wiki/#{ description.id }" }
expect(response).to have_http_status(:not_found)
end
@@ -73,6 +74,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do
let!(:other_tag) { create(:tag, primary_name: 'other_description_contract') }
let!(:other_version) do
TagDescriptionVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag_id: other_tag.id, language_code: 'ja', version_no: 1,
event_type: 'create', body: 'other tag history',
created_by_user_id: user.id, created_at: Time.current)
@@ -190,7 +192,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do
describe 'index' do
[nil, 'description_contract'].each do |title|
it "returns new content once in #{ title ? 'filtered' : 'unfiltered' } index" do
get '/wiki', params: { title: }
get '/wiki', params: title.nil? ? { } : { title: title }
expect(response).to have_http_status(:ok)
rows = json.select { |row| row.fetch('id') == description.id }
expect(rows.size).to eq(1)
+13 -1
ファイルの表示
@@ -13,7 +13,7 @@ RSpec.describe 'Tag description native API', type: :request do
def stored_state
[TagDescription, TagDescriptionVersion, TagName, TagVersion].map do |model|
model.order(:id).map(&:attributes)
end
end + [event_rows]
end
def fail_save model
@@ -91,6 +91,8 @@ RSpec.describe 'Tag description native API', type: :request do
let!(:version) do
TagDescriptionVersion.create!(
tag_id: tag.id, language_code: 'ja', body: 'native version', version_no: 1,
event_id: create_event!(
event_type: 'tag_descriptions.create', created_by_user: author).id,
event_type: 'create', created_by_user_id: author.id, created_at: Time.current)
end
let(:name_endpoint) { "/tags/name/#{ tag_name.name }/description" }
@@ -185,6 +187,7 @@ RSpec.describe 'Tag description native API', type: :request do
expect { native_write(body: 'created natively') }
.to change(TagDescription, :count).by(1)
.and change(TagDescriptionVersion, :count).by(1)
.and change { event_rows.length }.by(1)
expect(response).to have_http_status(:created)
description = TagDescription.find_by!(tag_id: tag.id, language_code: 'ja')
expect_current(description)
@@ -192,6 +195,10 @@ RSpec.describe 'Tag description native API', type: :request do
expect(version).to have_attributes(
body: 'created natively', version_no: 1, event_type: 'create',
created_by_user_id: editor.id)
event = event_rows.last
expect(event).to include(
'event_type' => 'tag_descriptions.create', 'created_by_user_id' => editor.id)
expect(version.event_id).to eq(event.fetch('id'))
get "/wiki/#{ description.id }"
expect(response).to have_http_status(:ok)
expect(json).to include('body' => description.body, 'revision_id' => version.id)
@@ -223,6 +230,7 @@ RSpec.describe 'Tag description native API', type: :request do
expect(description.version_no).to eq(1)
expect { native_write(body: 'after') }
.to change(TagDescriptionVersion, :count).by(1)
.and change { event_rows.length }.by(1)
expect(response).to have_http_status(:ok)
expect(description.reload).to have_attributes(
body: 'after', version_no: 2,
@@ -234,6 +242,10 @@ RSpec.describe 'Tag description native API', type: :request do
expect(version).to have_attributes(
tag_id: tag.id, language_code: 'ja', version_no: 2, event_type: 'update',
body: 'after', created_by_user_id: editor.id)
event = event_rows.last
expect(event).to include(
'event_type' => 'tag_descriptions.update', 'created_by_user_id' => editor.id)
expect(version.event_id).to eq(event.fetch('id'))
expect(revision.body).to eq(version.body)
expect(description_versions(page).order(:version_no).last.body).to eq(version.body)
get "/wiki/#{ description.id }"
+5 -1
ファイルの表示
@@ -121,10 +121,14 @@ RSpec.describe 'Tag group native API', type: :request do
[:create, :update].each do |operation|
it "#{ operation }s structured data and keeps unplaced membership" do
previous = group_version(group_id, 1)
native_write(operation)
expect { native_write(operation) }.to change { event_rows.length }.by(1)
expect(response).to have_http_status(operation == :create ? :created : :ok)
id = operation == :create ? json.fetch('id') : group_id
number = operation == :create ? 1 : 2
event = event_rows.last
event_type = operation == :create ? 'tag_groups.create' : 'tag_groups.update'
expect(event).to include('event_type' => event_type, 'created_by_user_id' => user.id)
expect(group_version(id, number).fetch('event_id')).to eq(event.fetch('id'))
expect_group_snapshot(
id, number:, members: snapshot_members, placements:, description: '全体の説明')
expect(group_version(group_id, 1)).to eq(previous)
+1
ファイルの表示
@@ -30,6 +30,7 @@ RSpec.describe 'TagVersions API', type: :request do
)
version =
TagVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag: tag,
version_no: version_no,
event_type: event_type,
+10
ファイルの表示
@@ -705,6 +705,7 @@ RSpec.describe 'Tags API', type: :request do
expect {
patch "/tags/#{tag.id}", params: { name: 'new_tag_name', category: 'meme' }
}.to change(TagVersion, :count).by(2)
.and change { event_rows.length }.by(1)
expect(response).to have_http_status(:ok)
@@ -719,6 +720,10 @@ RSpec.describe 'Tags API', type: :request do
expect(version_primary_name(versions.second, 'ja')).to eq('new_tag_name')
expect(versions.second.category).to eq('meme')
expect(versions.second.created_by_user_id).to eq(member_user.id)
event = event_rows.last
expect(event).to include(
'event_type' => 'tags.update', 'created_by_user_id' => member_user.id)
expect(versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
end
it 'updates deprecated state and records it in tag versions' do
@@ -1275,6 +1280,7 @@ RSpec.describe 'Tags API', type: :request do
deprecated: '0',
}
}.to change(TagVersion, :count).by(2)
.and change { event_rows.length }.by(1)
expect(response).to have_http_status(:ok)
@@ -1290,6 +1296,10 @@ RSpec.describe 'Tags API', type: :request do
expect(versions.second.category).to eq('meta')
expect(version_alias_names(versions.second, 'ja')).to include('spec_tag')
expect(versions.second.created_by_user_id).to eq(member_user.id)
event = event_rows.last
expect(event).to include(
'event_type' => 'tags.update', 'created_by_user_id' => member_user.id)
expect(versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
end
it '同じ tag の旧 alias へ戻しても rename できる' do
+323
ファイルの表示
@@ -0,0 +1,323 @@
require 'rails_helper'
require 'tmpdir'
require 'csv'
RSpec.describe 'Counts export contracts' do
include ActiveSupport::Testing::TimeHelpers
let(:payload) do
{ 'resource' => 'posts', 'definition' => 'history', 'mode' => 'details',
'locale' => 'ja', 'execution_at' => '2024-03-01T00:00:00Z',
'columns' => 'title', 'headers' => '["Title"]', 'format' => 'json' }
end
around do |example|
Dir.mktmpdir('counts-export-spec') do |path|
@counts_export_dir = Pathname.new(path)
HistoryAccess.set(user: nil) { example.run }
end
end
before do
allow(CountsExportDownload).to receive(:directory)
.and_return(@counts_export_dir)
end
context 'nico link IDs with the V1 export field name' do
let(:tag) { create(:tag, primary_name: 'linked export tag') }
let(:external) { create(:external_tag, name: 'external export tag') }
let(:payload) do
super().merge('resource' => 'nico_tags', 'columns' => 'linked_tags',
'headers' => '["linked_tags"]')
end
before do
NicoTagVersion.create!(external_tag: external, version_no: 1,
event_id: create_event!(event_type: 'unknown').id,
event_type: 'create', linked_tag_ids: '', created_at: Time.utc(2023))
NicoTagVersion.create!(external_tag: external, version_no: 2,
event_id: create_event!(event_type: 'unknown').id,
event_type: 'update', linked_tag_ids: tag.id.to_s, created_at: Time.utc(2024))
end
it 'exports recorded IDs with the legacy JSON key and metadata' do
result = JSON.parse(CountsTableExport.new(payload).body.to_a.join)
expect(result.fetch('data')).to eq([
{ 'linked_tags' => '' }, { 'linked_tags' => tag.id.to_s }])
expect(result.fetch('metadata').fetch('columns')).to eq(['linked_tags'])
keys = result.fetch('metadata').fetch('fields').map { _1.fetch('key') }
expect(keys).to include('linked_tags')
expect(keys).not_to include('linked_tag_ids')
end
it 'exports the selected legacy CSV header and ID values' do
body = CountsTableExport.new(payload.merge('format' => 'csv')).body.to_a.join
expect(CSV.parse(body.delete_prefix("\uFEFF")))
.to eq([['linked_tags'], [''], [tag.id.to_s]])
end
it 'keeps the legacy link column in default JSON exports' do
options = payload.except('columns', 'headers')
result = JSON.parse(CountsTableExport.new(options).body.to_a.join)
expect(result.fetch('metadata').fetch('columns')).to include('linked_tags')
result.fetch('data').each do |row|
expect(row).to have_key('linked_tags')
expect(row).not_to have_key('linked_tag_ids')
end
end
it 'rejects the physical column as a selected CSV or JSON field' do
['csv', 'json'].each do |format|
options = payload.merge('format' => format, 'columns' => 'linked_tag_ids')
expect { CountsTableExport.new(options) }.to raise_error(ArgumentError)
end
end
end
context 'serialization after the authorised query' do
let(:rows) { [{ title: "=formula\n\"quoted\"" }, { title: 'second' }] }
let(:page) do
{ count: 2, output_count: 2, visible_output_count: 2, page: 1, limit: 100,
versions: rows, execution_at: payload.fetch('execution_at') }
end
before do
query = CountsTableQuery.new(payload.slice(*CountsTableQuery::KEYS))
# SQL/authority are exercised in request specs; this unit boundary supplies their page.
allow(query).to receive(:each_page) do |before_page:, &block|
before_page.call
block.call(page)
end
allow(CountsTableQuery).to receive(:new).and_return(query)
end
it 'reports aggregation, actual output progress and regular-row metadata' do
progress = []
phases = []
body = CountsTableExport.new(payload).body(
progress: ->(done, total) { progress << [done, total] },
phase: ->(value) { phases << value }).to_a.join
result = JSON.parse(body)
expect(result['data']).to eq(rows.map(&:stringify_keys))
expect(result['metadata']).to include('logical_count_unit' => 'regular_result_rows',
'aggregate_records_in_logical_count' => false, 'output_count' => 2)
expect(phases).to eq(['aggregating', 'writing'])
expect(progress).to include([0, 2], [1, 2], [2, 2])
expect(progress.last).to eq([2, 2])
end
it 'preserves CSV multiline text and quotes while neutralising spreadsheet formulas' do
body = CountsTableExport.new(payload.merge('format' => 'csv')).body.to_a.join
expect(CSV.parse(body.delete_prefix("\uFEFF")))
.to eq([['Title'], ["'=formula\n\"quoted\""], ['second']])
end
it 'completes an isolated worker job with its real processed count and cancellable file' do
CountsExportJobs.pulse
token = CountsExportJobs.start(payload, 'owner', 'ja')
id = CountsExportJobs.authorised_id(token, 'owner')
state = CountsExportJobs.read(id).merge('status' => 'running', 'phase' => 'aggregating')
CountsExportJobs.write(state)
# Invoke the worker entrypoint directly; never fork or start an external worker in specs.
CountsExportJobs.perform(state)
ready = CountsExportJobs.status(token, 'owner')
expect(ready).to include('status' => 'ready', 'processed' => 2, 'total' => 2)
file = CountsExportDownload.find(ready.fetch('token'), 'owner')
expect(JSON.parse(File.read(file.fetch('path')))['data'].length).to eq(2)
CountsExportJobs.status(token, 'owner', cancel: true)
expect(CountsExportDownload.find(ready.fetch('token'), 'owner')).to be_nil
end
it 'marks a generation exception as failed without issuing a download token' do
CountsExportJobs.pulse
token = CountsExportJobs.start(payload, 'owner', 'ja')
id = CountsExportJobs.authorised_id(token, 'owner')
state = CountsExportJobs.read(id).merge('status' => 'running')
CountsExportJobs.write(state)
allow(CountsExportDownload).to receive(:prepare).and_raise(CountsExportDownload::TooLarge)
CountsExportJobs.perform(state)
expect(CountsExportJobs.status(token, 'owner')).to include('status' => 'failed')
expect(CountsExportJobs.status(token, 'owner')).not_to have_key('token')
end
end
it 'retains multilingual names and all inner rows in JSON with one selected display column' do
names = [{ language_code: 'ja', name: 'alpha', script_code: 'Jpan', aliases: [] },
{ language_code: 'ain', name: 'recorded', script_code: 'Kana',
aliases: [{ name: 'alias', script_code: 'Latn' }] }]
rows = [{ id: 1, name_ja: 'alpha', names: }, { id: 2, name_ja: 'beta', names: }]
projection = CountsExportProjection.new(payload.merge('columns' => 'tags',
'headers' => '["Tags"]', 'children' => 'tags.name_ja'),
resource: 'posts', summary: false, tags: false)
result = projection.project({ tags: rows })
expect(result['tags'].length).to eq(2)
expect(result['tags'].map { _1['name_ja'] }).to eq(['alpha', 'beta'])
expect(result['tags'].map { _1['names'] }).to eq([names, names])
end
it 'requires a live worker, enforces queue capacity and binds jobs to their owner' do
expect { CountsExportJobs.start(payload, 'owner', 'ja') }
.to raise_error(CountsExportJobs::WorkerUnavailable)
CountsExportJobs.pulse
token = CountsExportJobs.start(payload, 'owner', 'ja')
expect(CountsExportJobs.status(token, 'other')).to be_nil
expect(CountsExportJobs.status(token, 'owner')).to include(
'status' => 'queued', 'phase' => 'queued', 'processed' => 0, 'total' => nil)
(CountsExportDownload::MAX_CONCURRENT_EXPORTS - 1).times do
CountsExportJobs.start(payload, 'owner', 'ja')
end
expect { CountsExportJobs.start(payload, 'owner', 'ja') }
.to raise_error(CountsExportDownload::Busy)
expect(CountsExportJobs.status(token, 'owner', cancel: true)['status']).to eq('cancelled')
end
it 'fails stale queued/running jobs and preserves actual progress without a fake total' do
CountsExportJobs.pulse
token = CountsExportJobs.start(payload, 'owner', 'ja')
id = CountsExportJobs.authorised_id(token, 'owner')
state = CountsExportJobs.read(id)
state.merge!('status' => 'running', 'phase' => 'aggregating')
CountsExportJobs.write(state)
CountsExportJobs.checkpoint(id, 5, nil, phase: 'writing')
expect(CountsExportJobs.status(token, 'owner')).to include(
'phase' => 'writing', 'processed' => 5, 'total' => nil)
state = CountsExportJobs.read(id)
state['heartbeat_at'] = (Time.current - CountsExportJobs::HEARTBEAT_SECONDS - 1).to_f
CountsExportJobs.write(state)
expect(CountsExportJobs.status(token, 'owner')).to include(
'status' => 'failed', 'error' => 'worker_stalled', 'processed' => 5)
expect { CountsExportJobs.checkpoint(id, 6) }.to raise_error(CountsExportJobs::Cancelled)
end
it 'revokes an already generated download on a new discard and rejects wrong owners' do
exporter = instance_double(CountsExporter, resource: 'posts', format: 'json', body: ['{}'])
token = CountsExportDownload.prepare(exporter, 'owner')
expect(CountsExportDownload.find(token, 'owner')).not_to be_nil
expect(CountsExportDownload.find(token, 'other')).to be_nil
post = create(:post)
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post:, version_no: 1, event_type: 'discard', url: post.url,
title: 'private', tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024))
expect(CountsExportDownload.find(token, 'owner')).to be_nil
end
it 'rejects oversized output and removes its partial file' do
stub_const('CountsExportDownload::MAX_FILE_BYTES', 4)
exporter = instance_double(CountsExporter, resource: 'posts', format: 'json', body: ['12345'])
expect { CountsExportDownload.prepare(exporter, 'owner') }
.to raise_error(CountsExportDownload::TooLarge)
expect(Dir.glob(CountsExportDownload.directory.join('counts-*'))).to be_empty
end
it 'expires queued work after worker loss and rejects expired download signatures' do
CountsExportJobs.pulse
token = CountsExportJobs.start(payload, 'owner', 'ja')
travel_to(Time.current + CountsExportJobs::HEARTBEAT_SECONDS + 1) do
expect(CountsExportJobs.status(token, 'owner')).to include(
'status' => 'failed', 'error' => 'worker_unavailable')
end
exporter = instance_double(CountsExporter, resource: 'posts', format: 'json', body: ['{}'])
download = CountsExportDownload.prepare(exporter, 'owner')
travel_to(Time.current + CountsExportDownload::EXPIRY + 1) do
expect(CountsExportDownload.find(download, 'owner')).to be_nil
end
end
end
RSpec.describe 'Counts download acknowledgement' do
around do |example|
Dir.mktmpdir('counts-ack-spec') do |path|
@counts_ack_dir = Pathname.new(path)
HistoryAccess.set(user: nil) { example.run }
end
end
before do
allow(CountsExportDownload).to receive(:directory).and_return(@counts_ack_dir)
end
it 'binds an acknowledgement to the owner, job, file token and fresh attempt' do
CountsExportJobs.pulse
payload = { 'resource' => 'posts', 'definition' => 'history', 'mode' => 'details',
'format' => 'json', 'locale' => 'ja' }
token = CountsExportJobs.start(payload, 'owner', 'ja')
id = CountsExportJobs.authorised_id(token, 'owner')
state = CountsExportJobs.read(id).merge('status' => 'ready', 'token' => 'file-token')
CountsExportJobs.write(state)
receipt = CountsExportJobs.download_attempt(token, 'owner')
expect(receipt['download_status']).to eq('pending')
arguments = { file_token: 'file-token', attempt: receipt.fetch('download_attempt') }
expect(CountsExportJobs.download_attempt(token, 'other', **arguments)).to be_nil
expect(CountsExportJobs.download_attempt(token, 'owner', **arguments.merge(file_token: 'wrong'))).to be_nil
CountsExportJobs.download_attempt(token, 'owner', **arguments, status: 'streaming')
CountsExportJobs.download_attempt(token, 'owner', **arguments, status: 'started')
expect(CountsExportJobs.status(token, 'owner')['download_status']).to eq('started')
expect(CountsExportJobs.download_attempt(token, 'owner')['download_attempt'])
.to eq(receipt['download_attempt'])
CountsExportJobs.download_attempt(token, 'owner', **arguments, status: 'failed')
fresh = CountsExportJobs.download_attempt(token, 'owner')
expect(fresh['download_attempt']).not_to eq(receipt['download_attempt'])
expect(CountsExportJobs.download_attempt(token, 'owner', **arguments)).to be_nil
CountsExportJobs.status(token, 'owner', cancel: true)
expect(CountsExportJobs.download_attempt(token, 'owner')).to be_nil
end
end
RSpec.describe CountsExportDownload::ResponseBody do
it 'survives Rails buffer enumeration without closing before the first byte' do
Dir.mktmpdir('counts-body-spec') do |directory|
path = File.join(directory, 'file')
payload = 'data' * 20_000
File.binwrite(path, payload)
states = []
source = ActionDispatch::Response::FileBody.new(path)
body = described_class.new(source) { |state| states << state; true }
response = ActionDispatch::Response.new
response.body = body
_, _, rack_body = response.to_a
expect(states).to eq([])
chunks = []
rack_body.each { |chunk| chunks << chunk }
expect(chunks.join).to eq(payload)
expect(states.count('started')).to eq(1)
end
end
it 'acknowledges only after the first Rack yield and closes its bounded file body' do
Dir.mktmpdir('counts-body-spec') do |directory|
path = File.join(directory, 'file')
File.binwrite(path, 'x' * (64 * 1024 + 3))
states = []
source = ActionDispatch::Response::FileBody.new(path)
body = described_class.new(source) { |state| states << state; true }
chunks = []
body.each do |chunk|
expect(states).not_to include('started') if chunks.empty?
chunks << chunk
end
expect(chunks.join).to eq('x' * (64 * 1024 + 3))
expect(chunks.map(&:bytesize).max).to be <= 64 * 1024
expect(states.count('started')).to eq(1)
expect(body).not_to respond_to(:to_path)
expect { body.close }.not_to raise_error
end
end
it 'records a failed stream without converting an exception into a second HTTP response' do
Dir.mktmpdir('counts-body-spec') do |directory|
path = File.join(directory, 'file')
File.binwrite(path, 'data')
states = []
source = ActionDispatch::Response::FileBody.new(path)
body = described_class.new(source) { |state| states << state; true }
expect { body.each { raise IOError, 'connection closed' } }.to raise_error(IOError)
expect(states).to include('failed')
expect(states).not_to include('started')
expect { body.close }.not_to raise_error
end
end
end
+381
ファイルの表示
@@ -0,0 +1,381 @@
require 'rails_helper'
require 'csv'
RSpec.describe 'Counts masked pivot contracts' do
let(:admin) { create(:user, :admin) }
let(:member) { create(:user, :member) }
let(:guest) { create(:user, :guest) }
let(:params) do
{ 'resource' => 'posts', 'definition' => 'history', 'mode' => 'summary',
'locale' => 'ja', 'execution_at' => '2025-01-01T00:00:00Z',
'timezone' => 'UTC', 'rows' => 'tags', 'cols' => 'event_type', 'sort' => '' }
end
let!(:tags) do
['A', 'B', 'C'].map do |name|
tag = create(:tag, primary_name: name)
TagVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag:, version_no: 1, event_type: 'create', name:,
category: 'general', aliases: '', parent_tag_ids: '',
names: { ja: { name:, script_code: 'Jpan', auto_generated: false, aliases: [] } },
created_at: Time.utc(2023))
tag
end
end
before do
[[0, 1], [0], [1, 2]].each_with_index do |indexes, index|
post = create(:post)
snapshot = indexes.map { { tag_id: tags[_1].id, version_no: 1, sections: [] } }
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post:, version_no: 1, event_type: 'create',
title: "hidden-title-#{ index }", url: post.url, tags: '', tags_json: snapshot,
parent_post_ids: '', created_by_user: member, created_at: Time.utc(2024, 1))
next if index == 2
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post:, version_no: 2, event_type: 'discard',
title: "private-discard-#{ index }", url: post.url, tags: '', tags_json: [],
parent_post_ids: '', created_by_user: member, created_at: Time.utc(2024, 2))
end
end
def result(user, extra = {})
HistoryAccess.set(user:) { CountsTableQuery.new(params.merge(extra)).call }
end
def grand(result)
result.fetch(:pivot_totals).find { _1[:row].empty? && _1[:column].empty? }.fetch(:count)
end
it 'preserves monthly name subtotals and grand totals across pages' do
settings = { 'resource' => 'tags', 'rows' => 'created_at:month name' }
complete = result(admin, settings)
expect(complete[:group_count]).to eq(3)
expect(grand(complete)).to eq(3)
(1..3).each do |page|
paged = result(admin, settings.merge('limit' => '1', 'page' => page.to_s))
expect(grand(paged)).to eq(3)
subtotal = paged[:pivot_totals].find do |cell|
cell[:row] == ['2023-01'] && cell[:column].empty?
end
headers = paged[:row_headers].select { _1[:kind] == 'subtotal' }
if page == 3
expect(headers.map { _1[:values] }).to eq([['2023-01']])
expect(subtotal).not_to be_nil
expect(subtotal[:count]).to eq(3)
else
expect(headers).to eq([])
expect(subtotal).to be_nil
end
expect(paged[:cells].sum { _1[:count] }).to eq(1)
expect(paged[:group_count]).to eq(3)
end
end
it 'counts one opaque collection per private version across pages' do
[nil, guest, member, admin].each do |user|
page = result(user)
expected = user == admin ? 5 : 6
expect(grand(page)).to eq(expected)
expect(page[:cells].sum { _1[:count] }).to eq(expected)
expect(page).not_to have_key(:excluded_count)
paged = result(user, 'limit' => '1')
expect(grand(paged)).to eq(expected)
expect(result(user, 'limit' => '1', 'page' => '2')[:group_count])
.to eq(page[:group_count])
end
public_page = result(nil)
hidden = public_page[:cells].select { _1[:values].first == CountsAxes::MASKED_VALUE }
expect(hidden.sum { _1[:count] }).to eq(4)
expect(public_page[:axis_labels]['a0']).not_to have_key("internal:#{ tags.first.id }")
expect(public_page.to_json).not_to include('hidden-title', 'private-discard')
end
it 'retains distinct membership when tags are not a row axis' do
[nil, admin].each do |user|
expect(grand(result(user, 'rows' => 'event_type', 'cols' => 'tags')))
.to eq(user == admin ? 3 : 5)
expect(result(user, 'rows' => '', 'cols' => '')[:cells].sole[:count]).to eq(5)
details = result(user, 'mode' => 'details', 'rows' => '', 'cols' => '')
expect(details[:count]).to eq(user == admin ? 5 : 1)
end
end
it 'uses changed-record membership for totals and hides changes to private predecessors' do
post = PostVersion.order(:id).last.post
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post:, version_no: 2, event_type: 'update', title: 'changed-public-title',
url: post.url, tags: '',
tags_json: [{ tag_id: tags.last.id, version_no: 1, sections: [] }],
parent_post_ids: '', created_at: Time.utc(2024, 3))
restored = PostVersion.order(:id).first.post
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post: restored, version_no: 3, event_type: 'restore',
title: 'restored-public-title', url: restored.url, tags: '',
tags_json: [{ tag_id: tags.first.id, version_no: 1, sections: [] }], parent_post_ids: '',
created_at: Time.utc(2024, 4))
[nil, admin].each do |user|
page = result(user, 'rows' => 'tags changed_field', 'cols' => 'event_type')
expect(grand(page)).to eq(page[:cells].sum { _1[:count] })
expect(page[:cells]).not_to be_empty
if user.nil?
expect(page[:cells].any? { _1[:values][1] == CountsAxes::MASKED_VALUE }).to be(true)
end
expect(page.to_json).not_to include('hidden-title', 'private-discard')
end
end
it 'counts distinct changed records when tags are not additive row tuples' do
[nil, admin].each do |user|
page = result(user, 'rows' => 'changed_field', 'cols' => 'event_type')
expect(grand(page)).to eq(user == admin ? 2 : 4)
if user.nil?
expect(page[:cells].map { _1[:values].first }.uniq).to eq([CountsAxes::MASKED_VALUE])
end
# Private empty collections contribute the same opaque tuple as populated ones.
combined = result(user, 'rows' => 'changed_field', 'cols' => 'tags')
expect(grand(combined)).to eq(user == admin ? 0 : 4)
end
end
it 'keeps NULL as a public data value rather than treating it as a rollup' do
post = create(:post)
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post:, version_no: 1, event_type: 'create', title: 'public bot',
url: post.url, tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024, 5))
page = result(nil, 'rows' => 'user', 'cols' => 'event_type')
expect(page[:cells].select { _1[:values].first.nil? }.sum { _1[:count] }).to eq(1)
expect(grand(page)).to eq(6)
expect(page[:row_headers]).to include(hash_including(values: [nil], kind: 'data'))
end
it 'sorts public version numbers numerically including formerly hidden records' do
post = PostVersion.order(:id).last.post
[2, 10].each do |number|
PostVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
post:, version_no: number, event_type: 'update',
title: 'public numeric sort', url: post.url, tags: '', tags_json: [],
parent_post_ids: '', created_at: Time.utc(2024, 5))
end
page = result(nil, 'rows' => 'version_no', 'cols' => '')
values = page[:row_headers].select { _1[:kind] == 'data' }.map { _1[:values].first }
expect(values.map(&:to_i)).to eq([1, 2, 10])
end
it 'keeps public identity predicates and excludes private attributes' do
hidden_id = PostVersion.first.post_id.to_s
base = { 'rows' => '', 'cols' => '' }
condition = lambda do |field, operator, value|
{ 'conditions' => '', 'conditions.0.field' => field,
'conditions.0.operator' => operator, 'conditions.0.value' => value }
end
expect(result(nil, base.merge('resource_id' => hidden_id))[:cells].sole[:count]).to eq(2)
expect(result(nil, base.merge(condition.call('version_no', 'eq', '2')))[:cells]
.sole[:count]).to eq(2)
expect(result(nil, base.merge(condition.call('user', 'eq', member.id.to_s)))[:cells]
.sole[:count]).to eq(5)
expect(result(nil, base.merge(condition.call('title', 'eq', 'hidden-title-0')))[:cells]
.sole[:count]).to eq(0)
expect(result(nil, base.merge(condition.call('title', 'ne', 'hidden-title-0')))[:cells]
.sole[:count]).to eq(1)
expect(result(nil, base.merge(condition.call('title', 'eq', '********')))[:cells]
.sole[:count]).to eq(0)
expect(result(nil, base.merge(condition.call('video_ms', 'eq', '********')))[:cells]
.sole[:count]).to eq(0)
expect(result(nil, base.merge(condition.call('title', 'like', '%title%')))[:cells]
.sole[:count]).to eq(1)
expect(result(nil, base.merge(condition.call('title', 'like', '%')))[:cells]
.sole[:count]).to eq(1)
expect(result(nil, base.merge('tags' => 'C'))[:cells].sole[:count]).to eq(1)
expect(result(nil, base.merge('tags' => 'not:C'))[:cells].sole[:count]).to eq(0)
end
it 'retains public OR branches while rejecting private predicates on hidden rows' do
HistoryAccess.set(user: nil) do
population = CountsPopulation.new(params.merge('rows' => '', 'cols' => ''),
authorisation_flg: false)
condition = CountsCondition.new(population)
public_scope = condition.apply(population.scope,
{ 'field' => 'event_type', 'operator' => 'eq', 'value' => 'discard' })
private_scope = condition.apply(population.scope,
{ 'field' => 'title', 'operator' => 'ne', 'value' => 'unrelated' })
masked_scope = condition.apply(population.scope,
{ 'field' => 'title', 'operator' => 'eq', 'value' => '********' })
expect(public_scope.except(:select).count).to eq(2)
expect(public_scope.or(private_scope).except(:select).count).to eq(3)
expect(private_scope.or(public_scope).except(:select).count).to eq(3)
expect(public_scope.or(masked_scope).except(:select).count).to eq(2)
expect(public_scope.merge(private_scope).except(:select).count).to eq(0)
end
end
it 'excludes private collections from positive, negative, AND and OR tag searches' do
settings = { 'rows' => '', 'cols' => '' }
{ 'C' => 1, 'not:C' => 0, 'not:unknown' => 1, '********' => 0,
'not:********' => 1, 'not:C C' => 0, 'not:C type:or C' => 1 }.each do |value, count|
expect(result(nil, settings.merge('tags' => value))[:cells].sole[:count]).to eq(count)
end
end
it 'ignores empty numeric and interval bounds without restricting private history' do
settings = { 'rows' => '', 'cols' => '', 'conditions' => '' }
{ 'video_ms' => ['eq', 'ne', 'gt', 'ge', 'lt', 'le', 'range'],
'tags.sections' => ['range'] }.each do |field, operators|
operators.each do |operator|
condition = settings.merge('conditions.0.field' => field,
'conditions.0.operator' => operator)
empty = condition.merge('conditions.0.value' => '')
expect(result(nil, condition)[:cells].sole[:count]).to eq(5)
expect(result(nil, empty)[:cells].sole[:count]).to eq(5)
expect(result(nil, empty.merge('conditions.0.blank' => '0'))[:cells]
.sole[:count]).to eq(5)
if operator == 'range'
empty['conditions.0.end'] = ''
expect(result(nil, empty)[:cells].sole[:count]).to eq(5)
['conditions.0.value', 'conditions.0.end'].each do |key|
active = empty.merge(key => '100')
expect(result(nil, active)[:cells].sole[:count]).to eq(0)
end
end
end
end
end
it 'keeps typed marker sets active after incompatible marker values are removed' do
settings = { 'rows' => '', 'cols' => '', 'conditions' => '',
'conditions.0.field' => 'video_ms', 'conditions.0.values' => '1',
'condition_values' => '', 'condition_values.0.condition' => '0',
'condition_values.0.value_type' => 'string',
'condition_values.0.value' => '********' }
['in', 'not_in'].each do |operator|
expected = operator == 'in' ? 0 : 1
condition = settings.merge('conditions.0.operator' => operator)
expect(result(nil, condition)[:cells].sole[:count]).to eq(expected)
end
end
it 'restricts active numeric comparisons and preserves empty selection semantics' do
settings = { 'rows' => '', 'cols' => '', 'conditions' => '',
'conditions.0.field' => 'video_ms' }
['eq', 'ne', 'gt', 'ge', 'lt', 'le', 'range'].each do |operator|
condition = settings.merge('conditions.0.operator' => operator,
'conditions.0.value' => '0')
condition['conditions.0.end'] = '100' if operator == 'range'
# The one public row is NULL; private NULLs and real values never match.
expected = operator == 'ne' ? 1 : 0
expect(result(nil, condition)[:cells].sole[:count]).to eq(expected)
end
['in', 'not_in'].each do |operator|
condition = settings.merge('conditions.0.operator' => operator,
'conditions.0.values' => '1', 'condition_values' => '')
expected = operator == 'not_in' ? 5 : 0
expect(result(nil, condition)[:cells].sole[:count]).to eq(expected)
end
blank = settings.merge('conditions.0.operator' => 'eq', 'conditions.0.blank' => '1')
expect(result(nil, blank)[:cells].sole[:count]).to eq(1)
end
it 'preserves empty tag sets and restricts active blank collection predicates' do
settings = { 'rows' => '', 'cols' => '', 'conditions' => '',
'conditions.0.field' => 'tags' }
['any', 'all', 'none'].each do |operator|
condition = settings.merge('conditions.0.operator' => operator,
'conditions.0.values' => '1', 'condition_values' => '')
expected = operator == 'any' ? 0 : 5
expect(result(nil, condition)[:cells].sole[:count]).to eq(expected)
end
blank = settings.merge('conditions.0.operator' => 'eq', 'conditions.0.blank' => '1')
expect(result(nil, blank)[:cells].sole[:count]).to eq(0)
end
it 'rejects server-only coordinates in column order' do
probes = [
{ 'cols' => 'tags', 'column_order' => '', 'column_order.0.value_type' => 'string',
'column_order.0.value' => CountsAxes::MASKED_VALUE }]
HistoryAccess.set(user: nil) do
probes.each do |probe|
expect { CountsTableQuery.new(params.merge(probe)).call }
.to raise_error(CountsAxes::PrivateCondition)
end
end
end
it 'keeps private collection and change counts independent of stored elements' do
settings = [
{ 'rows' => 'tags', 'cols' => 'changed_field' },
{ 'rows' => 'changed_field tags', 'cols' => '' }]
original = settings.map { result(nil, _1) }
snapshot = tags.map do |tag|
{ tag_id: tag.id, version_no: 1, sections: [] }
end
hidden_posts = PostVersion.where(event_type: 'discard').pluck(:post_id)
PostVersion.where(post_id: hidden_posts).update_all(
title: 'same hidden value', tags_json: snapshot)
expect(settings.map { result(nil, _1) }).to eq(original)
end
it 'excludes private tag and change coordinates from drill conditions' do
['tags', 'changed_field'].each do |field|
settings = { 'rows' => field, 'cols' => '' }
page = result(nil, settings)
cell = page[:cells].find { _1[:values] == [CountsAxes::MASKED_VALUE] }
expect(cell[:count]).to eq(4)
[CountsAxes::MASKED_VALUE, CountsAxes::MASKED_TEXT].each do |value|
drill = { 'drill' => '', 'drill.0.field' => field,
'drill.0.value_type' => 'string', 'drill.0.value' => value }
expect(result(nil, settings.merge(drill))[:cells]).to be_empty
details = result(nil, settings.merge(drill).merge('mode' => 'details'))
expect(details[:versions]).to be_empty
condition = { 'conditions' => '', 'conditions.0.field' => field,
'conditions.0.operator' => 'axis_eq',
'conditions.0.value_type' => 'string', 'conditions.0.value' => value }
expect(result(nil, settings.merge(condition))[:cells]).to be_empty
end
end
end
it 'rejects real private tag and change coordinates for non-admin users' do
[nil, guest, member].each do |user|
{ 'tags' => "internal:#{ tags.first.id }", 'changed_field' => 'title' }.each do
|field, value|
settings = { 'rows' => '', 'cols' => '', 'drill' => '',
'drill.0.field' => field, 'drill.0.value_type' => 'string',
'drill.0.value' => value }
expect(result(user, settings)[:cells].sole[:count]).to eq(0)
expect(result(admin, settings)[:cells].sole[:count]).to be > 0
end
end
end
it 'keeps public column sorting usable while guessed private coordinates match nothing' do
first = result(nil, 'rows' => 'event_type', 'cols' => 'tags')
probe = result(nil, 'rows' => 'event_type', 'cols' => 'tags',
'column_order' => '', 'column_order.0.value_type' => 'string',
'column_order.0.value' => "internal:#{ tags.first.id }")
expect(probe[:column_values]).to eq(first[:column_values])
expect(probe[:cells]).to eq(first[:cells])
expect(probe[:column_values_editable]).to be(false)
expect(probe.to_json).not_to include("internal:#{ tags.first.id }")
end
it 'exports masked coordinates and safe CSV labels from the same pivot' do
HistoryAccess.set(user: nil) do
payload = params.merge('format' => 'json')
json = JSON.parse(CountsTableExport.new(payload).body.to_a.join)
expect(json['data'].sum { _1['kind'] == 'data' ? _1['count'] : 0 }).to eq(6)
expect(json.to_json).not_to include('hidden-title', 'private-discard')
csv = CountsTableExport.new(payload.merge('format' => 'csv')).body.to_a.join
expect(csv).to include('********')
expect(csv).not_to include(CountsAxes::MASKED_VALUE, 'hidden-title', 'private-discard')
end
end
end
+2
ファイルの表示
@@ -56,6 +56,7 @@ RSpec.describe MaterialSyncImporter do
profile: 'legacy_drive',
export_path: '素材/a.png',
created_by_user: user)
events = event_rows
result =
described_class.import!(
@@ -73,6 +74,7 @@ RSpec.describe MaterialSyncImporter do
expect(result.action).to eq(:unchanged)
expect(material.reload.tag).to eq(tag)
expect(material.url).to eq('https://example.com/human')
expect(event_rows).to eq(events)
end
it 'suppresses a Google Drive candidate by relative path prefix' do
+41
ファイルの表示
@@ -14,6 +14,47 @@ RSpec.describe MaterialSyncRunner do
let(:drive_client) { instance_double(GoogleDrive::ApiClient) }
describe '#sync!' do
it 'groups created and updated materials across sync sources into one event' do
existing = Material.create!(
source_kind: 'google_drive_file', source_file_id: 'existing-file',
source_path: 'old.png', url: 'https://example.com/existing',
created_by_user: user, updated_by_user: user)
source
MaterialSyncSource.create!(
name: 'Second Drive source', source_kind: 'google_drive_path',
source_file_id: 'folder-456', profile: 'legacy_drive',
created_by_user: user, updated_by_user: user)
allow(GoogleDrive::ApiClient).to receive(:new).and_return(drive_client)
allow(drive_client).to receive(:each_material_file_under_folder) do |folder_id, &block|
id = folder_id == 'folder-123' ? 'existing-file' : 'new-file'
block.call(id:, name: "#{ id }.png", mime_type: 'image/png',
relative_path: "素材/#{ id }.png", sha256_checksum: nil,
web_view_link: "https://drive.google.com/file/d/#{ id }/view")
end
allow(drive_client).to receive(:download_to_tempfile) do |id, **|
Tempfile.new(['sync-event', '.png']).tap do |file|
file.binmode
file.write(id)
file.rewind
end
end
allow(MaterialThumbnailGenerator).to receive(:generate!)
previous_ids = MaterialVersion.pluck(:id)
result = nil
expect { result = described_class.sync_enabled! }
.to change { event_rows.length }.by(1)
expect(result).to have_attributes(imported: 1, updated: 1, failed: 0)
event = event_rows.last
expect(event.fetch('event_type')).to eq('materials.sync')
created = Material.find_by!(source_file_id: 'new-file')
versions = MaterialVersion.where.not(id: previous_ids)
expect(versions.pluck(:material_id).uniq).to contain_exactly(existing.id, created.id)
expect(versions.pluck(:event_type)).to include('create', 'update')
expect(versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
end
it 'imports google drive path candidates as they are yielded' do
first_entry = { id: 'file-1',
name: 'a.png',
+23 -8
ファイルの表示
@@ -8,22 +8,29 @@ RSpec.describe NicoTagVersionRecorder do
described_class.record!(external_tag:, event_type:, created_by_user: member)
end
it 'records the external association and platform-qualified name' do
it 'records the external association and derives the platform-qualified name' do
version = record(:create)
expect(version).to have_attributes(
external_tag:, version_no: 1, event_type: 'create',
name: 'nico:raw tag[]', linked_tags: '', created_by_user: member)
name: 'nico:raw tag[]', linked_tag_ids: '', created_by_user: member)
expect(external_tag.reload.nico_tag_versions).to contain_exactly(version)
expect(version.event).to have_attributes(
event_type: 'external_tags.create', created_by_user: member)
expect(version.event.nico_tag_versions).to contain_exactly(version)
end
it 'uses the latest history number when the record has no version_no column' do
first = record(:create)
external_tag.update!(name: 'changed')
tag = create(:tag)
NicoTagRelation.create!(nico_tag: external_tag, tag:)
second = record(:update)
expect(second).to have_attributes(version_no: 2, name: 'nico:changed')
expect(second).to have_attributes(version_no: 2, name: 'nico:raw tag[]',
linked_tag_ids: tag.id.to_s)
expect(first.reload.name).to eq('nico:raw tag[]')
expect(second.event).to have_attributes(
event_type: 'external_tags.update', created_by_user: member)
expect(external_tag.reload.has_attribute?(:version_no)).to be(false)
end
@@ -40,7 +47,7 @@ RSpec.describe NicoTagVersionRecorder do
expect(version).to have_attributes(
external_tag: external, name: 'registered_external:foo', version_no: 1,
event_type: 'create', linked_tags: '', created_by_user: member)
event_type: 'create', linked_tag_ids: '', created_by_user: member)
end
it 'returns the latest version without appending an unchanged snapshot' do
@@ -64,13 +71,21 @@ RSpec.describe NicoTagVersionRecorder do
expect(external_tag.nico_tag_versions).to contain_exactly(first)
end
it 'records sorted linked internal names and later link removal' do
it 'rejects discard and restore events for external tags' do
[:discard, :restore].each do |event_type|
expect { record(event_type) }
.to raise_error(ArgumentError, "Invalid event_type: #{ event_type }")
end
end
it 'records sorted linked internal IDs and later link removal' do
tags = ['z_link', 'a_link'].map { |name| create(:tag, primary_name: name, category: :general) }
tags.each { |tag| NicoTagRelation.create!(nico_tag: external_tag, tag:) }
expect(record(:create).linked_tags).to eq('a_link z_link')
expect(record(:create).linked_tag_ids).to eq(tags.map(&:id).sort.join(' '))
external_tag.linked_tags = [tags.first]
expect(record(:update)).to have_attributes(version_no: 2, linked_tags: 'z_link')
expect(record(:update)).to have_attributes(version_no: 2, linked_tag_ids: tags.first.id.to_s)
end
end
+83
ファイルの表示
@@ -88,3 +88,86 @@ RSpec.describe PostBulkCreator do
expect(creator).to have_received(:create!).with(locale).once
end
end
RSpec.describe 'PostBulkCreator event grouping', type: :model do
self.use_transactional_tests = false
before do
expect(Rails.env.test?).to be(true)
expect(ApplicationRecord.connection.open_transactions).to eq(0)
@prefix = "bulk_event_#{ SecureRandom.hex(8) }"
@actor = create(:user, :member)
allow(PostCreatePreflight).to receive(:new) do |attributes:, **|
specs = [{ name: attributes.fetch(:title), category: :general }]
instance_double(PostCreatePreflight, run: attributes.merge(
existing_post_id: nil, snapshot_tag_specs: specs, post_tag_specs: specs,
normalised_parent_post_ids: [], tag_sections: { }, video_ms: nil))
end
end
after do
if @actor
posts = Post.where(uploaded_user: @actor)
PostTagSection.where(post_id: posts.select(:id)).delete_all
PostTag.where(post_id: posts.select(:id)).delete_all
PostVersion.where(post_id: posts.select(:id)).delete_all
posts.delete_all
tag_ids = TagName.where(name: ["#{ @prefix }_1", "#{ @prefix }_2"]).pluck(:tag_id)
TagVersion.where(tag_id: tag_ids).delete_all
TagName.where(tag_id: tag_ids).delete_all
Tag.where(id: tag_ids).delete_all
EventRecords::EventRow.where(created_by_user_id: @actor.id).delete_all
@actor.delete
end
end
it 'shares one creation event between real workers and their tag histories' do
posts = [1, 2].map do |number|
{ title: "#{ @prefix }_#{ number }",
url: "https://example.com/#{ @prefix }/#{ number }" }
end
result = nil
expect {
result = PostBulkCreator.new(actor: @actor, posts:, thumbnails: { })
.run(Locale.nipponese)
}.to change { event_rows.length }.by(1)
expect(result.fetch(:results).map { _1.fetch(:status) }).to eq(['created', 'created'])
event = event_rows.last
expect(event).to include(
'event_type' => 'posts.create', 'created_by_user_id' => @actor.id)
versions = PostVersion.where(created_by_user_id: @actor.id)
expect(versions.count).to eq(2)
expect(versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
tag_versions = TagVersion.where(created_by_user_id: @actor.id)
expect(tag_versions.count).to eq(2)
expect(tag_versions.pluck(:event_id).uniq).to eq([event.fetch('id')])
end
it 'does not leave an event when every row fails validation' do
preflight = instance_double(PostCreatePreflight)
allow(PostCreatePreflight).to receive(:new).and_return(preflight)
allow(preflight).to receive(:run).and_raise(
PostCreatePreflight::ValidationFailed.new(fields: { url: ['invalid'] }))
events = event_rows
result = PostBulkCreator.new(
actor: @actor, posts: [{ title: 'invalid' }], thumbnails: { }).run(Locale.nipponese)
expect(result.fetch(:results).map { _1.fetch(:status) }).to eq(['failed'])
expect(event_rows).to eq(events)
end
it 'does not leave an event when every row is skipped' do
preflight = instance_double(PostCreatePreflight, run: { existing_post_id: 123 })
allow(PostCreatePreflight).to receive(:new).and_return(preflight)
events = event_rows
result = PostBulkCreator.new(
actor: @actor, posts: [{ title: 'existing' }], thumbnails: { }).run(Locale.nipponese)
expect(result.fetch(:results).map { _1.fetch(:status) }).to eq(['skipped'])
expect(event_rows).to eq(events)
end
end
+33
ファイルの表示
@@ -0,0 +1,33 @@
require 'rails_helper'
RSpec.describe QueryState do
it 'decodes typed axis values and keeps absence distinct from an empty set' do
{ 'boolean' => ['false', false], 'number' => ['7', 7],
'string' => ['a b & c', 'a b & c'], 'null' => ['', nil] }.each do |type, (raw, expected)|
params = { 'conditions' => '', 'conditions.0.field' => 'user',
'conditions.0.operator' => 'axis_eq',
'conditions.0.value_type' => type, 'conditions.0.value' => raw }
expect(described_class.records(params, 'conditions').sole['value']).to eq(expected)
end
expect(described_class.records({}, 'conditions')).to eq([])
expect(described_class.records({ 'conditions' => '' }, 'conditions')).to eq([])
expect(described_class.records({ 'conditions' => '', 'conditions.0.values' => '1' },
'conditions')).to eq([{ 'values' => true }])
end
it 'rejects gaps, unknown keys, JSON, wrong types and oversized lists' do
[
{ 'conditions' => '[]' }, { 'conditions' => '', 'conditions.1.field' => 'user' },
{ 'conditions' => '', 'conditions.0.sql' => 'TRUE' },
{ 'conditions' => '', 'conditions.0.blank' => 'true' },
{ 'conditions' => '', 'conditions.0.value' => ['x'] },
{ 'drill' => '', 'drill.0.field' => 'user', 'drill.0.value_type' => 'object' },
{ 'conditions' => 'x' * 131_073 }].each do |params|
expect { described_class.validate!(params, ['conditions', 'drill']) }
.to raise_error(ArgumentError)
end
['user user', 'user title', ' user', (1..201).to_a.join(' ')].each do |value|
expect { described_class.list(value) }.to raise_error(ArgumentError)
end
end
end
+28 -2
ファイルの表示
@@ -8,10 +8,15 @@ RSpec.describe 'Shared resource IDs across MySQL connections', type: :model do
expect(Rails.env.test?).to be(true)
expect(ApplicationRecord.connection.adapter_name).to match(/mysql/i)
expect(ApplicationRecord.connection_pool.size).to be >= 3
@main_connection = ApplicationRecord.connection
# A leaked fixture pin makes worker threads share the main session. Fail before
# starting a worker rather than diagnosing IS_USED_LOCK(NULL) after a timeout.
expect(@main_connection.open_transactions).to eq(0)
@prefix = "shared_ids_#{ SecureRandom.hex(8) }"
@tag = create(:tag, primary_name: "#{ @prefix }_owner")
@group = TagGroup.create!(name: "#{ @prefix }_owner", version_no: 1)
@tag_ids = Queue.new
@event_ids = Queue.new
@threads = []
@release = Queue.new
end
@@ -38,17 +43,27 @@ RSpec.describe 'Shared resource IDs across MySQL connections', type: :model do
ids << @tag.id if @tag
TagName.where(tag_id: ids).delete_all
Tag.where(id: ids).delete_all
event_ids = []
event_ids << @event_ids.pop until @event_ids.empty? if @event_ids
EventRecords::EventRow.where(id: event_ids).delete_all
end
end
def await_queue queue
Timeout.timeout(15) { queue.pop }
value = Timeout.timeout(15) { queue.pop }
raise value if value.is_a?(Exception)
value
end
def worker &block
Thread.new do
Thread.current.report_on_exception = false
ApplicationRecord.connection_pool.with_connection(&block)
ApplicationRecord.connection_pool.with_connection do |connection|
raise 'Shared ID worker reused the main connection' if connection.equal?(@main_connection)
block.call(connection)
end
end.tap { @threads << _1 }
end
@@ -61,11 +76,17 @@ RSpec.describe 'Shared resource IDs across MySQL connections', type: :model do
when :description
TagDescription.create!(tag: @tag, language_code: 'ja', body: '', version_no: 1)
when :description_version
event = create_event!(event_type: 'unknown')
@event_ids << event.id
TagDescriptionVersion.create!(
event_id: event.id,
tag_id: @tag.id, language_code: 'ja', body: 'body', version_no: 1,
event_type: 'create', created_at: Time.current)
when :group_version
event = create_event!(event_type: 'unknown')
@event_ids << event.id
TagGroupVersion.create!(
event_id: event.id,
tag_group: @group, name: @group.name, version_no: 1, event_type: 'create',
members: [], placements: [], created_at: Time.current)
when :group
@@ -86,11 +107,16 @@ RSpec.describe 'Shared resource IDs across MySQL connections', type: :model do
ActiveSupport::Notifications.subscribed(subscriber, 'sql.active_record') do
id = create_resource(kind).id
end
raise 'Shared ID creation did not acquire a named lock' unless lock_name
ready << [id, connection.select_value('SELECT CONNECTION_ID()').to_i, lock_name]
await_queue(@release)
raise ActiveRecord::Rollback if rollback
end
id
rescue StandardError => error
ready << error
raise
end
[thread, await_queue(ready)]
end
+3
ファイルの表示
@@ -15,10 +15,12 @@ RSpec.describe SharedResourceIdAllocator do
TagGroup.new(id:, name: "group #{ number }", version_no: number)
when 'tag_description_versions'
TagDescriptionVersion.new(
event_id: create_event!(event_type: 'unknown').id,
id:, tag_id: tag.id, language_code: 'ja', body: 'body', version_no: number,
event_type: 'create', created_at: Time.current)
when 'tag_group_versions'
TagGroupVersion.new(
event_id: create_event!(event_type: 'unknown').id,
id:, tag_group_id: group.id, name: group.name, version_no: number,
event_type: 'create', members: [], placements: [], created_at: Time.current)
end
@@ -82,6 +84,7 @@ RSpec.describe SharedResourceIdAllocator do
it 'reserves a Tag ID referenced only by remaining description history' do
high = [Tag.maximum(:id).to_i, ExternalTag.maximum(:id).to_i].max + 1000
TagDescriptionVersion.create!(
event_id: create_event!(event_type: 'unknown').id,
tag_id: high, language_code: 'ja', version_no: 1, body: 'history',
event_type: 'create', created_at: Time.current)
expect(create(:tag).id).to be > high
+108
ファイルの表示
@@ -18,6 +18,9 @@ RSpec.describe VersionRecorder do
expect(version.version_no).to eq(1)
expect(post_record.reload.version_no).to eq(1)
expect(version.event).to have_attributes(
event_type: 'posts.create', created_by_user: member)
expect(version.event.created_at).to be_present
end
it 'updates record version_no when creating the next version' do
@@ -36,6 +39,8 @@ RSpec.describe VersionRecorder do
expect(version.version_no).to eq(2)
expect(post_record.reload.version_no).to eq(2)
expect(version.event).to have_attributes(
event_type: 'posts.update', created_by_user: member)
end
it 'does not create a new version or advance version_no when snapshot is unchanged' do
@@ -44,6 +49,7 @@ RSpec.describe VersionRecorder do
post: post_record,
event_type: :create,
created_by_user: member)
events = Event.count
expect {
version =
@@ -56,6 +62,108 @@ RSpec.describe VersionRecorder do
}.not_to change(PostVersion, :count)
expect(post_record.reload.version_no).to eq(1)
expect(Event.count).to eq(events)
end
it 'retains an explicitly supplied event' do
event = Event.create!(event_type: 'posts.sync', reason: 'scheduled sync')
expect {
version = PostVersionRecorder.record!(
post: post_record, event_type: :create, created_by_user: nil, event:)
expect(version.event).to eq(event)
}.not_to change(Event, :count)
end
it 'shares an operation event between nested tag and post histories' do
tag = create(:tag, primary_name: 'operation_tag')
version = nil
tag_version = nil
expect {
described_class.record_operation!(
event_type: 'posts.create', created_by_user: member, reason: 'new post') do
described_class.record_operation!(event_type: 'tags.create', created_by_user: nil) do
tag_version = TagVersionRecorder.record!(
tag:, event_type: :create, created_by_user: member)
end
version = PostVersionRecorder.record!(
post: post_record, event_type: :create, created_by_user: member)
end
}.to change(Event, :count).by(1)
expect(version.event_id).to eq(tag_version.event_id)
expect(version.event).to have_attributes(
event_type: 'posts.create', created_by_user: member, reason: 'new post')
expect(described_class.current_event).to be_nil
end
it 'does not persist an event for an unchanged operation' do
PostVersionRecorder.record!(
post: post_record, event_type: :create, created_by_user: member)
expect {
described_class.record_operation!(event_type: 'posts.update', created_by_user: member) do
PostVersionRecorder.record!(
post: post_record.reload, event_type: :update, created_by_user: member)
end
}.not_to change(Event, :count)
end
it 'rolls back histories and their event when an operation fails' do
record = post_record
events = Event.count
versions = PostVersion.count
expect {
described_class.record_operation!(
event_type: 'posts.create', created_by_user: member, savepoint_flg: true) do
PostVersionRecorder.record!(post: record, event_type: :create, created_by_user: member)
raise 'operation failed'
end
}.to raise_error('operation failed')
expect(Event.count).to eq(events)
expect(PostVersion.count).to eq(versions)
expect(record.reload.version_no).to eq(1)
expect(described_class.current_event).to be_nil
version = PostVersionRecorder.record!(post: record, event_type: :create, created_by_user: nil)
expect(version.event).to have_attributes(event_type: 'posts.create', created_by_user: nil)
end
it 'isolates operation context between threads' do
described_class.with_event(event_type: 'posts.sync', created_by_user: nil) do
event = described_class.current_event
expect(Thread.new { described_class.current_event }.value).to be_nil
expect(described_class.current_event).to equal(event)
end
expect(described_class.current_event).to be_nil
end
it 'reuses a batch event after its first transaction rolls back' do
record = post_record
version = nil
expect {
described_class.with_event(event_type: 'posts.sync', created_by_user: nil) do
event = described_class.current_event
described_class.record_operation!(
event_type: 'posts.create', created_by_user: nil, savepoint_flg: true) do
PostVersionRecorder.record!(post: record, event_type: :create, created_by_user: nil)
raise ActiveRecord::Rollback
end
expect(event).to be_new_record
version = PostVersionRecorder.record!(
post: record.reload, event_type: :create, created_by_user: nil)
expect(version.event).to equal(event)
end
}.to change(Event, :count).by(1)
expect(version.event.event_type).to eq('posts.sync')
expect(record.post_versions).to contain_exactly(version)
end
it 'raises when record version_no is older than the latest version' do

変更されたファイルが多すぎるため、一部のファイルは表示されません さらに表示