コミットを比較

...
28 コミット
作成者 SHA1 メッセージ 日付
Codex Night Worker f21cd7d671 AI: address issue #483 2026-11-18 12:28:16 +09:00
Codex Night Worker 9c71a9fc02 AI: address issue #483 2026-11-17 04:09:38 +09:00
Codex Night Worker cb1f947a36 AI: address issue #483 2026-11-16 13:11:57 +09:00
みてるぞとCodex Night Worker dda3d87073 外部タグの履歴管理方式を変更 (#445) (#480)
Co-authored-by: Codex Night Worker <codex-night@localhost>
Reviewed-on: #480
Co-authored-by: miteruzo <miteruzo@naver.com>
2026-11-15 16:18:27 +09:00
テツノニジカとCodex Night Worker e13a52db9a AI: #454 広場のタグ検索,スマホ版だとサゼストが途中で見切れる問題 (#456)
Refs #454

## Automated implementation (requires human review)
[SidebarComponent.tsx](frontend/src/components/layout/SidebarComponent.tsx) の1行を修正しました。`overflow-y-auto` を `md` 以上に限定し、スマホでサゼストがサイドバーの領域外で切られる原因を取り除きました。

差分を確認済みです。指定に従い、テスト・lint・ビルドは未実行です。実画面でのスマホ表示とPC表示の確認も未実施です。

## Automated review (not a guarantee of correctness)
```text
The change restricts sidebar scrolling to medium and larger viewports. No actionable regression was identified in the current diff.

```

## Verification
- Tests/builds/migrations were not authorized and were not invoked by the controller.
- Codex was instructed not to run them; its command transcript should be inspected.
- Do not merge before manual acceptance and required checks.

---------

Co-authored-by: Codex Night Worker <codex-night@localhost>
Reviewed-on: #456
Co-authored-by: テツノニジカ <969+codex-bot@noreply.localhost>
2026-11-14 17:09:38 +09:00
みてるぞ f05d8a4b3a 集計で独裁者以外がほとんど検索条件を指定できないバグ (#444) (#452)
Reviewed-on: #452
Co-authored-by: miteruzo <miteruzo@naver.com>
2026-11-09 00:03:26 +09:00
みてるぞ dbae864f24 広場のタグ検索がスマホで利かない問題 (#435) (#453)
Reviewed-on: #453
Co-authored-by: miteruzo <miteruzo@naver.com>
2026-11-08 14:52:38 +09:00
みてるぞ 8d00a6545e API の処理に時間がかかりすぎる問題 (#442) (#449)
Reviewed-on: #449
Co-authored-by: miteruzo <miteruzo@naver.com>
2026-11-07 19:01:20 +09:00
みてるぞ f8747a0f74 Merge pull request '集計機能追加 (#434)' (#441) from feature/434 into main
Reviewed-on: #441
2026-11-01 19:39:04 +09:00
みてるぞ f874d6e405 #434 2026-11-01 19:34:52 +09:00
みてるぞ 22e57564a5 #434 2026-11-01 18:18:54 +09:00
みてるぞ 29002d5393 #434 2026-11-01 04:22:26 +09:00
みてるぞ 8a4c1d1220 #434 2026-11-01 01:12:29 +09:00
みてるぞ 74c5242828 #434 2026-11-01 00:20:32 +09:00
みてるぞ 6afbab8eee #434 2026-10-31 23:30:45 +09:00
みてるぞ 14853a8c5f #434 2026-10-31 20:44:28 +09:00
みてるぞ 4f3990feb6 #434 2026-10-31 19:07:14 +09:00
みてるぞ 5b77285a6b #434 2026-10-31 14:25:31 +09:00
みてるぞ dba409c3e8 #434 2026-10-31 13:50:10 +09:00
みてるぞ 6a0f1ec819 #434 2026-10-31 04:08:05 +09:00
みてるぞ efbd640963 #434 2026-10-29 23:31:04 +09:00
みてるぞ 7fc2faeaf0 #434 2026-10-29 22:26:49 +09:00
みてるぞ 237331481f #434 2026-10-29 21:38:38 +09:00
みてるぞ e857a33ec3 #434 2026-10-29 19:49:05 +09:00
みてるぞ 9f92dbfb06 #434 2026-10-29 18:10:13 +09:00
みてるぞ 549a45e9d5 #434 2026-10-29 12:38:03 +09:00
みてるぞ 7090f63ab6 #434 2026-10-29 12:30:42 +09:00
みてるぞ 7536330ecf #434 2026-10-29 12:28:20 +09:00
134個のファイルの変更、10775行の追加、1177行の削除
-35
ファイルの表示
@@ -1,35 +0,0 @@
## 背景
なぜ必要か。
## 対象範囲
- backend:
- frontend:
- docs:
- migration:
## やること
- [ ]
## 受け入れ条件
- [ ]
## 実行すべき確認
- [ ] `cd backend && bundle exec rspec`
- [ ] `cd frontend && npm run build`
- [ ] `cd frontend && npm run lint`
## 禁止事項
- unrelated refactor はしない
- 既存 API response shape を壊さない
- 認証・認可・BAN を弱めない
## Codex への指示
この issue を読んで実装してください。
不明点があれば、実装前に調査結果と選択肢を提示してください。
+2
ファイルの表示
@@ -71,3 +71,5 @@ gem "rspec-rails", "~> 8.0", :groups => [:development, :test]
gem 'aws-sdk-s3', require: false
gem 'rails-i18n', '~> 8.0.0'
gem "json", "~> 2.19.9"
+142 -134
ファイルの表示
@@ -1,29 +1,29 @@
GEM
remote: https://rubygems.org/
specs:
actioncable (8.0.2)
actionpack (= 8.0.2)
activesupport (= 8.0.2)
actioncable (8.0.5.1)
actionpack (= 8.0.5.1)
activesupport (= 8.0.5.1)
nio4r (~> 2.0)
websocket-driver (>= 0.6.1)
zeitwerk (~> 2.6)
actionmailbox (8.0.2)
actionpack (= 8.0.2)
activejob (= 8.0.2)
activerecord (= 8.0.2)
activestorage (= 8.0.2)
activesupport (= 8.0.2)
actionmailbox (8.0.5.1)
actionpack (= 8.0.5.1)
activejob (= 8.0.5.1)
activerecord (= 8.0.5.1)
activestorage (= 8.0.5.1)
activesupport (= 8.0.5.1)
mail (>= 2.8.0)
actionmailer (8.0.2)
actionpack (= 8.0.2)
actionview (= 8.0.2)
activejob (= 8.0.2)
activesupport (= 8.0.2)
actionmailer (8.0.5.1)
actionpack (= 8.0.5.1)
actionview (= 8.0.5.1)
activejob (= 8.0.5.1)
activesupport (= 8.0.5.1)
mail (>= 2.8.0)
rails-dom-testing (~> 2.2)
actionpack (8.0.2)
actionview (= 8.0.2)
activesupport (= 8.0.2)
actionpack (8.0.5.1)
actionview (= 8.0.5.1)
activesupport (= 8.0.5.1)
nokogiri (>= 1.8.5)
rack (>= 2.2.4)
rack-session (>= 1.0.1)
@@ -31,35 +31,35 @@ GEM
rails-dom-testing (~> 2.2)
rails-html-sanitizer (~> 1.6)
useragent (~> 0.16)
actiontext (8.0.2)
actionpack (= 8.0.2)
activerecord (= 8.0.2)
activestorage (= 8.0.2)
activesupport (= 8.0.2)
actiontext (8.0.5.1)
actionpack (= 8.0.5.1)
activerecord (= 8.0.5.1)
activestorage (= 8.0.5.1)
activesupport (= 8.0.5.1)
globalid (>= 0.6.0)
nokogiri (>= 1.8.5)
actionview (8.0.2)
activesupport (= 8.0.2)
actionview (8.0.5.1)
activesupport (= 8.0.5.1)
builder (~> 3.1)
erubi (~> 1.11)
rails-dom-testing (~> 2.2)
rails-html-sanitizer (~> 1.6)
activejob (8.0.2)
activesupport (= 8.0.2)
activejob (8.0.5.1)
activesupport (= 8.0.5.1)
globalid (>= 0.3.6)
activemodel (8.0.2)
activesupport (= 8.0.2)
activerecord (8.0.2)
activemodel (= 8.0.2)
activesupport (= 8.0.2)
activemodel (8.0.5.1)
activesupport (= 8.0.5.1)
activerecord (8.0.5.1)
activemodel (= 8.0.5.1)
activesupport (= 8.0.5.1)
timeout (>= 0.4.0)
activestorage (8.0.2)
actionpack (= 8.0.2)
activejob (= 8.0.2)
activerecord (= 8.0.2)
activesupport (= 8.0.2)
activestorage (8.0.5.1)
actionpack (= 8.0.5.1)
activejob (= 8.0.5.1)
activerecord (= 8.0.5.1)
activesupport (= 8.0.5.1)
marcel (~> 1.0)
activesupport (8.0.2)
activesupport (8.0.5.1)
base64
benchmark (>= 0.3)
bigdecimal
@@ -92,22 +92,22 @@ GEM
aws-sigv4 (~> 1.5)
aws-sigv4 (1.12.1)
aws-eventstream (~> 1, >= 1.0.2)
base64 (0.2.0)
base64 (0.3.0)
bcrypt_pbkdf (1.1.1)
bcrypt_pbkdf (1.1.1-arm64-darwin)
bcrypt_pbkdf (1.1.1-x86_64-darwin)
benchmark (0.4.0)
bigdecimal (3.1.9)
benchmark (0.5.0)
bigdecimal (4.1.3)
bootsnap (1.18.6)
msgpack (~> 1.2)
brakeman (7.0.2)
racc
builder (3.3.0)
chronic (0.10.2)
concurrent-ruby (1.3.5)
connection_pool (2.5.3)
crass (1.0.6)
date (3.4.1)
concurrent-ruby (1.3.8)
connection_pool (3.0.2)
crass (1.0.7)
date (3.6.0)
diff-lcs (1.6.2)
discard (1.4.0)
activerecord (>= 4.2, < 9.0)
@@ -115,8 +115,9 @@ GEM
dotenv-rails (3.1.8)
dotenv (= 3.1.8)
railties (>= 6.1)
drb (2.2.1)
drb (2.2.3)
ed25519 (1.4.0)
erb (6.0.7)
erubi (1.13.1)
factory_bot (6.5.6)
activesupport (>= 6.1.0)
@@ -134,7 +135,7 @@ GEM
gemojione (4.3.3)
json
github-markup (4.0.2)
globalid (1.2.1)
globalid (1.4.0)
activesupport (>= 6.1)
gollum (6.1.0)
gemojione (~> 4.1)
@@ -166,19 +167,20 @@ GEM
gollum-rugged_adapter (3.0)
mime-types (~> 3.4)
rugged (~> 1.5)
i18n (1.14.7)
i18n (1.15.2)
concurrent-ruby (~> 1.0)
image_processing (1.14.0)
mini_magick (>= 4.9.5, < 6)
ruby-vips (>= 2.0.17, < 3)
io-console (0.8.0)
irb (1.15.2)
io-console (0.9.4)
irb (1.18.0)
pp (>= 0.6.0)
prism (>= 1.3.0)
rdoc (>= 4.0.0)
reline (>= 0.4.2)
jmespath (1.6.2)
json (2.12.0)
jwt (2.10.1)
json (2.19.9)
jwt (3.3.0)
base64
kamal (2.6.1)
activesupport (>= 7.0)
@@ -198,15 +200,16 @@ GEM
language_server-protocol (3.17.0.5)
lint_roller (1.1.0)
logger (1.7.0)
loofah (2.24.1)
loofah (2.25.2)
crass (~> 1.0.2)
nokogiri (>= 1.12.0)
mail (2.8.1)
mail (2.9.1)
logger
mini_mime (>= 0.1.1)
net-imap
net-pop
net-smtp
marcel (1.0.4)
marcel (1.2.1)
mime-types (3.7.0)
logger
mime-types-data (~> 3.2025, >= 3.2025.0507)
@@ -215,45 +218,46 @@ GEM
benchmark
logger
mini_mime (1.1.5)
minitest (5.25.5)
msgpack (1.8.0)
multi_json (1.15.0)
minitest (6.0.6)
drb (~> 2.0)
prism (~> 1.5)
msgpack (1.8.5)
multi_json (1.21.2)
mustache (1.1.1)
mustache-sinatra (2.0.0)
mustache (~> 1.0)
mustermann (3.0.3)
ruby2_keywords (~> 0.0.1)
mustermann (3.1.1)
mysql2 (0.5.6)
net-imap (0.5.8)
net-imap (0.6.8)
date
net-protocol
net-pop (0.1.2)
net-protocol
net-protocol (0.2.2)
net-protocol (0.4.0)
timeout
net-scp (4.1.0)
net-ssh (>= 2.6.5, < 8.0.0)
net-sftp (4.0.0)
net-ssh (>= 5.0.0, < 8.0.0)
net-smtp (0.5.1)
net-smtp (0.5.2)
net-protocol
net-ssh (7.3.0)
nio4r (2.7.4)
nokogiri (1.18.8-aarch64-linux-gnu)
nio4r (2.7.5)
nokogiri (1.19.4-aarch64-linux-gnu)
racc (~> 1.4)
nokogiri (1.18.8-aarch64-linux-musl)
nokogiri (1.19.4-aarch64-linux-musl)
racc (~> 1.4)
nokogiri (1.18.8-arm-linux-gnu)
nokogiri (1.19.4-arm-linux-gnu)
racc (~> 1.4)
nokogiri (1.18.8-arm-linux-musl)
nokogiri (1.19.4-arm-linux-musl)
racc (~> 1.4)
nokogiri (1.18.8-arm64-darwin)
nokogiri (1.19.4-arm64-darwin)
racc (~> 1.4)
nokogiri (1.18.8-x86_64-darwin)
nokogiri (1.19.4-x86_64-darwin)
racc (~> 1.4)
nokogiri (1.18.8-x86_64-linux-gnu)
nokogiri (1.19.4-x86_64-linux-gnu)
racc (~> 1.4)
nokogiri (1.18.8-x86_64-linux-musl)
nokogiri (1.19.4-x86_64-linux-musl)
racc (~> 1.4)
octicons (19.15.2)
ostruct (0.6.1)
@@ -261,70 +265,73 @@ GEM
parser (3.3.8.0)
ast (~> 2.4.1)
racc
pp (0.6.2)
pp (0.6.4)
prettyprint
prettyprint (0.2.0)
prism (1.4.0)
psych (5.2.6)
prettyprint (0.2.1)
prism (1.9.0)
psych (5.5.0)
date
stringio
puma (6.6.0)
nio4r (~> 2.0)
racc (1.8.1)
rack (3.1.14)
rack (3.2.7)
rack-cors (2.0.2)
rack (>= 2.0.0)
rack-protection (4.1.1)
rack-protection (4.2.1)
base64 (>= 0.1.0)
logger (>= 1.6.0)
rack (>= 3.0.0, < 4)
rack-session (2.1.1)
rack-session (2.1.2)
base64 (>= 0.1.0)
rack (>= 3.0.0)
rack-test (2.2.0)
rack-test (2.3.0)
rack (>= 1.3)
rackup (2.2.1)
rackup (2.3.1)
rack (>= 3)
rails (8.0.2)
actioncable (= 8.0.2)
actionmailbox (= 8.0.2)
actionmailer (= 8.0.2)
actionpack (= 8.0.2)
actiontext (= 8.0.2)
actionview (= 8.0.2)
activejob (= 8.0.2)
activemodel (= 8.0.2)
activerecord (= 8.0.2)
activestorage (= 8.0.2)
activesupport (= 8.0.2)
rails (8.0.5.1)
actioncable (= 8.0.5.1)
actionmailbox (= 8.0.5.1)
actionmailer (= 8.0.5.1)
actionpack (= 8.0.5.1)
actiontext (= 8.0.5.1)
actionview (= 8.0.5.1)
activejob (= 8.0.5.1)
activemodel (= 8.0.5.1)
activerecord (= 8.0.5.1)
activestorage (= 8.0.5.1)
activesupport (= 8.0.5.1)
bundler (>= 1.15.0)
railties (= 8.0.2)
rails-dom-testing (2.2.0)
railties (= 8.0.5.1)
rails-dom-testing (2.3.0)
activesupport (>= 5.0.0)
minitest
nokogiri (>= 1.6)
rails-html-sanitizer (1.6.2)
loofah (~> 2.21)
rails-html-sanitizer (1.7.1)
loofah (~> 2.25, >= 2.25.2)
nokogiri (>= 1.15.7, != 1.16.7, != 1.16.6, != 1.16.5, != 1.16.4, != 1.16.3, != 1.16.2, != 1.16.1, != 1.16.0.rc1, != 1.16.0)
rails-i18n (8.0.2)
i18n (>= 0.7, < 2)
railties (>= 8.0.0, < 9)
railties (8.0.2)
actionpack (= 8.0.2)
activesupport (= 8.0.2)
railties (8.0.5.1)
actionpack (= 8.0.5.1)
activesupport (= 8.0.5.1)
irb (~> 1.13)
rackup (>= 1.0.0)
rake (>= 12.2)
thor (~> 1.0, >= 1.2.2)
tsort (>= 0.2)
zeitwerk (~> 2.6)
rainbow (3.1.1)
rake (13.2.1)
rdoc (6.13.1)
rake (13.4.2)
rdoc (6.17.0)
erb
psych (>= 4.0.0)
regexp_parser (2.10.0)
reline (0.6.1)
tsort
regexp_parser (2.13.1)
reline (0.7.0)
io-console (~> 0.5)
rexml (3.4.1)
rexml (3.4.4)
rouge (3.30.0)
rspec-core (3.13.6)
rspec-support (~> 3.13.0)
@@ -345,20 +352,20 @@ GEM
rspec-support (3.13.6)
rss (0.3.1)
rexml
rubocop (1.75.6)
json (~> 2.3)
rubocop (1.92.0)
json (>= 2.3)
language_server-protocol (~> 3.17.0.2)
lint_roller (~> 1.1.0)
parallel (~> 1.10)
parallel (>= 1.10)
parser (>= 3.3.0.2)
rainbow (>= 2.2.2, < 4.0)
regexp_parser (>= 2.9.3, < 3.0)
rubocop-ast (>= 1.44.0, < 2.0)
regexp_parser (>= 2.11.3, < 3.0)
rubocop-ast (>= 1.49.0, < 2.0)
ruby-progressbar (~> 1.7)
unicode-display_width (>= 2.4.0, < 4.0)
rubocop-ast (1.44.1)
rubocop-ast (1.50.0)
parser (>= 3.3.7.2)
prism (~> 1.4)
prism (~> 1.7)
rubocop-performance (1.25.0)
lint_roller (~> 1.1)
rubocop (>= 1.75.0, < 2.0)
@@ -377,24 +384,23 @@ GEM
ruby-vips (2.2.3)
ffi (~> 1.12)
logger
ruby2_keywords (0.0.5)
rugged (1.9.0)
securerandom (0.4.1)
sinatra (4.1.1)
sinatra (4.2.1)
logger (>= 1.6.0)
mustermann (~> 3.0)
rack (>= 3.0.0, < 4)
rack-protection (= 4.1.1)
rack-protection (= 4.2.1)
rack-session (>= 2.0.0, < 3)
tilt (~> 2.0)
sinatra-contrib (4.1.1)
sinatra-contrib (4.2.1)
multi_json (>= 0.0.2)
mustermann (~> 3.0)
rack-protection (= 4.1.1)
sinatra (= 4.1.1)
rack-protection (= 4.2.1)
sinatra (= 4.2.1)
tilt (~> 2.0)
sprockets (4.2.2)
concurrent-ruby (~> 1.0)
sprockets (4.4.1)
concurrent-ruby (~> 1.1)
logger
rack (>= 2.2.4, < 4)
sprockets-helpers (1.4.0)
@@ -403,14 +409,14 @@ GEM
actionpack (>= 6.1)
activesupport (>= 6.1)
sprockets (>= 3.0.0)
sqlite3 (2.6.0-aarch64-linux-gnu)
sqlite3 (2.6.0-aarch64-linux-musl)
sqlite3 (2.6.0-arm-linux-gnu)
sqlite3 (2.6.0-arm-linux-musl)
sqlite3 (2.6.0-arm64-darwin)
sqlite3 (2.6.0-x86_64-darwin)
sqlite3 (2.6.0-x86_64-linux-gnu)
sqlite3 (2.6.0-x86_64-linux-musl)
sqlite3 (2.10.1-aarch64-linux-gnu)
sqlite3 (2.10.1-aarch64-linux-musl)
sqlite3 (2.10.1-arm-linux-gnu)
sqlite3 (2.10.1-arm-linux-musl)
sqlite3 (2.10.1-arm64-darwin)
sqlite3 (2.10.1-x86_64-darwin)
sqlite3 (2.10.1-x86_64-linux-gnu)
sqlite3 (2.10.1-x86_64-linux-musl)
sshkit (1.24.0)
base64
logger
@@ -418,18 +424,19 @@ GEM
net-sftp (>= 2.1.2)
net-ssh (>= 2.8.0)
ostruct
stringio (3.1.7)
stringio (3.2.0)
therubyrhino (2.1.2)
therubyrhino_jar (>= 1.7.4, < 1.7.9)
therubyrhino_jar (1.7.8)
thor (1.3.2)
thor (1.5.0)
thruster (0.1.13)
thruster (0.1.13-aarch64-linux)
thruster (0.1.13-arm64-darwin)
thruster (0.1.13-x86_64-darwin)
thruster (0.1.13-x86_64-linux)
tilt (2.6.0)
timeout (0.4.3)
tilt (2.9.0)
timeout (0.6.1)
tsort (0.2.0)
twitter-text (1.14.7)
unf (~> 0.1.0)
tzinfo (2.0.6)
@@ -440,16 +447,16 @@ GEM
unicode-display_width (3.1.4)
unicode-emoji (~> 4.0, >= 4.0.4)
unicode-emoji (4.0.4)
uri (1.0.3)
uri (1.1.2)
useragent (0.16.11)
webrick (1.9.1)
websocket-driver (0.7.7)
websocket-driver (0.8.2)
base64
websocket-extensions (>= 0.1.0)
websocket-extensions (0.1.5)
whenever (1.0.0)
chronic (>= 0.6.3)
zeitwerk (2.7.2)
zeitwerk (2.8.3)
PLATFORMS
aarch64-linux
@@ -473,6 +480,7 @@ DEPENDENCIES
factory_bot_rails
gollum
image_processing (~> 1.14)
json (~> 2.19.9)
jwt
kamal
mysql2 (~> 0.5.6)
+3
ファイルの表示
@@ -27,6 +27,9 @@ class ApplicationController < ActionController::API
private
def enforce_query_contract!
if request.query_parameters.keys.any? { _1 == 'tag_refs' || _1.start_with?('tag_refs.') }
return head :bad_request
end
pairs = URI.decode_www_form(request.query_string)
return head :bad_request if pairs.length > 1000 || pairs.map(&:first).uniq.length != pairs.length
return head :bad_request if request.query_parameters.values.any? { !_1.is_a?(String) }
+109 -9
ファイルの表示
@@ -12,6 +12,15 @@ class CountsExportsController < ApplicationController
render json: { error: 'export_worker_unavailable' }, status: :service_unavailable
rescue CountsExportDownload::Busy
head :too_many_requests
rescue CountsAxes::TooManyAxes
render json: { error: 'too_many_axes', max_axes: CountsAxes::MAX_AXES }, status: :bad_request
rescue CountsAxes::PrivateCondition
render json: { error: 'private_condition' }, status: :forbidden
rescue CountsTableQuery::TooManyColumns
render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS },
status: :unprocessable_entity
rescue ActiveRecord::StatementTimeout
render json: { error: 'counts_query_timeout' }, status: :service_unavailable
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
head :bad_request
end
@@ -24,6 +33,14 @@ class CountsExportsController < ApplicationController
job_state(cancel: true)
end
def start_download
result = CountsExportJobs.download_attempt(params[:token], session[:counts_export_owner])
return head :not_found unless result
response.headers['Cache-Control'] = 'private, no-store'
render json: result
end
def prepare
locale = resolve_locale!
payload = request.request_parameters.merge(request.query_parameters)
@@ -40,11 +57,25 @@ class CountsExportsController < ApplicationController
head :gateway_timeout
rescue CountsExportDownload::StorageFull, Errno::ENOSPC
head :insufficient_storage
rescue CountsAxes::TooManyAxes
render json: { error: 'too_many_axes', max_axes: CountsAxes::MAX_AXES }, status: :bad_request
rescue CountsAxes::PrivateCondition
render json: { error: 'private_condition' }, status: :forbidden
rescue CountsTableQuery::TooManyColumns
render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS },
status: :unprocessable_entity
rescue ActiveRecord::StatementTimeout
render json: { error: 'counts_query_timeout' }, status: :service_unavailable
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
head :bad_request
end
def download
if params[:job].present? || params[:attempt].present?
return head :method_not_allowed unless request.get?
return confirmed_download
end
file = CountsExportDownload.find(params[:token], session[:counts_export_owner])
return head :not_found unless file
@@ -55,15 +86,6 @@ class CountsExportsController < ApplicationController
disposition: 'attachment'
end
def job_state(cancel:)
result = CountsExportJobs.status(params[:token], session[:counts_export_owner], cancel:)
return head :not_found unless result
response.headers['Cache-Control'] = 'private, no-store'
render json: result
end
private :job_state
def index
locale = resolve_locale!
payload = request.request_parameters.except('authenticity_token')
@@ -79,7 +101,85 @@ class CountsExportsController < ApplicationController
filename: "counts-#{ exporter.resource }.#{ exporter.format }",
type: exporter.format == 'json' ? 'application/json' : 'text/csv',
disposition: 'attachment'
rescue CountsAxes::TooManyAxes
render json: { error: 'too_many_axes', max_axes: CountsAxes::MAX_AXES }, status: :bad_request
rescue CountsAxes::PrivateCondition
render json: { error: 'private_condition' }, status: :forbidden
rescue CountsTableQuery::TooManyColumns
render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS },
status: :unprocessable_entity
rescue ActiveRecord::StatementTimeout
render json: { error: 'counts_query_timeout' }, status: :service_unavailable
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
head :bad_request
end
private
def confirmed_download
owner = session[:counts_export_owner]
arguments = { file_token: params[:token], attempt: params[:attempt] }
if (!(arguments[:attempt].present?) ||
!(CountsExportJobs.download_attempt(params[:job], owner, **arguments)))
return head :not_found
end
file = CountsExportDownload.find(params[:token], owner)
unless file
CountsExportJobs.download_attempt(params[:job], owner, **arguments, status: 'unavailable')
return head :not_found
end
job = params[:job]
authority = HistoryAccess.user
first_check = true
state = lambda do |status|
if status == 'active'
if first_check
first_check = false
fresh = authority && User.find_by(id: authority.id)
next false if authority && (!fresh || fresh.banned?)
authority = fresh
HistoryAccess.set(user: authority) do
CountsExportDownload.find(arguments[:file_token], owner) &&
CountsExportJobs.stream_active?(job, owner, **arguments)
end
else
CountsExportJobs.stream_active?(job, owner, **arguments)
end
else
HistoryAccess.set(user: authority) do
CountsExportJobs.download_attempt(job, owner, **arguments, status:)
end
end
end
unless CountsExportJobs.download_attempt(job, owner, **arguments, status: 'streaming')
return head :conflict
end
response.headers['Cache-Control'] = 'private, no-store'
response.headers['Last-Modified'] = File.mtime(file.fetch('path')).httpdate
response.headers['X-Accel-Buffering'] = 'no'
send_file file.fetch('path'),
filename: "counts-#{ file.fetch('resource') }.#{ file.fetch('format') }",
type: file['format'] == 'json' ? 'application/json' : 'text/csv',
disposition: 'attachment'
# Keep Rails' bounded FileBody reader, but hide to_path from Rack::Sendfile:
# offloading would bypass the GET acknowledgement and cancellation checks.
stream = CountsExportDownload::ResponseBody.new(response.stream, &state)
self.response_body = stream
rescue IOError, SystemCallError
stream&.close
CountsExportJobs.download_attempt(params[:job], owner, **arguments, status: 'failed')
raise if response.committed?
head :service_unavailable
end
def job_state cancel:
result = CountsExportJobs.status(params[:token], session[:counts_export_owner], cancel:)
return head :not_found unless result
response.headers['Cache-Control'] = 'private, no-store'
render json: result
end
end
+9
ファイルの表示
@@ -18,9 +18,18 @@ class CountsTablesController < ApplicationController
def index
resolve_locale!
render json: CountsTableQuery.new(request.query_parameters).call
rescue CountsAxes::TooManyAxes
render json: { error: 'too_many_axes', max_axes: CountsAxes::MAX_AXES }, status: :bad_request
rescue CountsAxes::PrivateCondition
render json: { error: 'private_condition' }, status: :forbidden
rescue CountsTableQuery::TooManyColumns
render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS },
status: :unprocessable_entity
rescue CountsPivotTotals::TimedOut
Rails.logger.warn('counts_query_timeout: pivot_totals')
render json: { error: 'counts_query_timeout', stage: 'totals',
timeout_seconds: CountsTableQuery::SQL_TIMEOUT_SECONDS },
status: :service_unavailable
rescue ActiveRecord::StatementTimeout => error
Rails.logger.warn("counts_query_timeout: #{ error.class }")
render json: { error: 'counts_query_timeout' }, status: :service_unavailable
+7 -12
ファイルの表示
@@ -14,8 +14,9 @@ class PostVersionsController < ApplicationController
offset = (page - 1) * limit
q = HistoryAccess.visible(version_scope)
q = HistoryAccess.visible(PostVersion.all)
q = q.where('post_versions.post_id = ?', post_id) if post_id
q = q.with_previous_version if tag_id || external_tag_id
if external_tag_id || (tag_id && !(Tag.exists?(id: tag_id)))
q = q.where('JSON_CONTAINS(post_versions.tags_json,' +
"JSON_OBJECT('external_tag_id', #{ external_tag_id || tag_id })) " +
@@ -28,9 +29,12 @@ class PostVersionsController < ApplicationController
count = q.except(:select, :order, :limit, :offset).count
versions = q.order(Arel.sql('post_versions.created_at DESC, post_versions.id DESC'))
ids = q.order(Arel.sql('post_versions.created_at DESC, post_versions.id DESC'))
.limit(limit)
.offset(offset)
.pluck('post_versions.id')
versions = version_scope.merge(q).where(id: ids)
.order(Arel.sql('post_versions.created_at DESC, post_versions.id DESC'))
render json: { versions: serialise_versions(versions, locale.language_code), count:, population_count: HistoryAccess.population_count(PostVersion) }
end
@@ -96,18 +100,9 @@ class PostVersionsController < ApplicationController
def external_tag_names_for snapshots
ids = snapshots.flatten.filter_map { _1['external_tag_id'] }.uniq
names = ExternalTag.where(id: ids).pluck(:id, :platform, :name).to_h { |id, platform, name|
ExternalTag.where(id: ids).pluck(:id, :platform, :name).to_h { |id, platform, name|
[id, "#{ platform }:#{ name }"]
}
missing_ids = ids - names.keys
HistoryAccess.visible(NicoTagVersion.all)
.where(tag_id: missing_ids)
.order(:tag_id, version_no: :desc)
.pluck(:tag_id, :name)
.each { |id, name| names[id] ||= name }
names
end
def snapshot_tag_literals snapshots, tag_names, external_tag_names
+60 -63
ファイルの表示
@@ -37,20 +37,10 @@ class PostsController < ApplicationController
offset = (page - 1) * limit
pt_max_sql =
HistoryAccess.visible(PostVersion.all)
.select('post_id, MAX(created_at) AS max_updated_at')
.group('post_id')
.to_sql
updated_at_all_sql =
'GREATEST(posts.updated_at,' +
'COALESCE(pt_max.max_updated_at, posts.updated_at))'
needs_history_join = order[0] == 'updated_at' || updated_between.any?(&:present?)
q =
filtered_posts(locale)
.joins("LEFT JOIN (#{ pt_max_sql }) pt_max ON pt_max.post_id = posts.id")
.reselect('posts.*', Arel.sql("#{ updated_at_all_sql } AS updated_at_all"))
.preload(:external_tags, :uploaded_user, :parents, :children,
post_tags: [:sections, { tag: [:deerjikists, :materials,
:tag_names, :tag_descriptions] }])
@@ -66,12 +56,25 @@ class PostsController < ApplicationController
end
q = q.where('posts.created_at >= ?', created_between[0]) if created_between[0]
q = q.where('posts.created_at <= ?', created_between[1]) if created_between[1]
count_scope = q
if needs_history_join
history_times = HistoryAccess.visible(PostVersion.all)
.select('post_id, MAX(created_at) AS latest_at').group(:post_id)
q = q.joins("LEFT JOIN (#{ history_times.to_sql }) history_times " +
'ON history_times.post_id = posts.id')
updated_at_all_sql = 'GREATEST(posts.updated_at, ' +
'COALESCE(history_times.latest_at, posts.updated_at))'
end
if updated_between[0]
q = q.where("#{ updated_at_all_sql } >= ?", updated_between[0])
end
if updated_between[1]
q = q.where("#{ updated_at_all_sql } <= ?", updated_between[1])
end
count_scope = q if updated_between.any?(&:present?)
if needs_history_join
q = q.select('posts.*', "#{ updated_at_all_sql } AS updated_at")
end
sort_sql =
case order[0]
@@ -88,12 +91,20 @@ class PostsController < ApplicationController
.limit(limit)
.offset(offset)
.to_a
latest_version_times =
if needs_history_join
{ }
else
HistoryAccess.visible(PostVersion.where(post_id: posts.map(&:id)))
.group(:post_id).maximum(:created_at)
end
q = q.except(:select, :order)
count_scope = count_scope.except(:select, :order)
render json: { posts: posts.map { |post|
updated_at = [post.updated_at, latest_version_times[post.id]].compact.max
PostRepr.base(post, language_code: locale.language_code)
.merge(updated_at: post.updated_at_all).tap do |json|
.merge('updated_at' => updated_at).tap do |json|
json['thumbnail'] =
if post.thumbnail.attached?
rails_storage_proxy_url(post.thumbnail, only_path: false)
@@ -101,7 +112,7 @@ class PostsController < ApplicationController
nil
end
end
}, count: q.group_values.present? ? q.count.size : q.count }
}, count: count_scope.group_values.present? ? count_scope.count.size : count_scope.count }
end
def random
@@ -409,52 +420,40 @@ class PostsController < ApplicationController
target = Locale.find_by(code:, deprecated_at: nil)
return head :not_found unless target
literals = params[:tags].to_s.split
references = tag_filter_references(literals)
names = literals.each_with_index.map do |literal, index|
reference = references&.[](index)
(reference && reference['negative'] == false) ? literal : literal.sub(/\Anot:/i, '')
end
internal = references ? {} : TagName.where(
language_code: source.language_code, name: names).pluck(:name, :tag_id).to_h
canonical = references ? {} : TagName.where(
language_code: source.language_code, primary_flg: true,
tag_id: internal.values).pluck(:tag_id, :name).to_h
external_names = names.map { canonical[internal[_1]] || _1 }
external = references ? {} : ExternalTag.where(
"CONCAT(platform, ':', name) IN (?)", external_names).pluck(
Arel.sql("CONCAT(platform, ':', name)"), :id).to_h
ids = references ? references.filter_map { _1['tag_id'] } : internal.values
translated = TagName.where(language_code: target.language_code,
primary_flg: true, tag_id: ids).pluck(:tag_id, :name).to_h
tokens = params[:tags].to_s.split
literals = PostTagSearch.literals(source, tokens.join(' '))
source_ids = PostTagSearch.references_for(source, literals)
ids = source_ids.values.flat_map(&:first).uniq
names = TagName.where(language_code: target.language_code,
primary_flg: true, tag_id: ids).pluck(:tag_id, :name).to_h
filters = literals.map do |literal|
internal, = source_ids.fetch(literal[:name])
name = internal.empty? ? literal[:name] : names[internal.first]
return head :unprocessable_entity unless name && internal.length <= 1
filters = literals.each_with_index.map do |literal, index|
name = names[index]
reference = if references
references[index].except('name')
else
matches = {}
matches['tag_id'] = internal[name] if internal[name]
external_id = external[external_names[index]]
matches['external_tag_id'] = external_id if external_id
matches.empty? ? { 'missing' => true } : matches
end
name = translated[reference['tag_id']] || name
negative = reference.fetch('negative', literal.downcase.start_with?('not:'))
prefix = negative ? literal[/\Anot:/i] || 'not:' : ''
reference.merge('name' => "#{ prefix }#{ name }", 'negative' => negative)
literal[:negative] ? "not:#{ name }" : name
end
render json: { references: filters }
translated = PostTagSearch.literals(target, filters.join(' '))
target_ids = PostTagSearch.references_for(target, translated)
literals.zip(translated).each do |literal, translation|
unless (source_ids.fetch(literal[:name]).map(&:sort) ==
target_ids.fetch(translation[:name]).map(&:sort))
return head :unprocessable_entity
end
end
index = 0
filters = tokens.map do |token|
next token if token.downcase == 'type:or'
value = filters[index]
index += 1
value
end
render json: { tags: filters.join(' ') }
end
private
def tag_filter_references literals
PostTagSearch.parse_references(params.to_unsafe_h, literals)
rescue ArgumentError, JSON::ParserError, TypeError
raise ActionController::BadRequest
end
def filtered_posts locale
tag_names = params[:tags].to_s.split
match_type = params[:match]
@@ -466,19 +465,17 @@ class PostsController < ApplicationController
end
def filter_posts_by_tags locale, tag_names, match_type
references = tag_filter_references(tag_names)
literals = PostTagSearch.literals(locale, tag_names.join(' '), references:)
literals = PostTagSearch.literals(locale, tag_names.join(' '))
return Post.all if literals.empty?
if match_type == 'any'
literals.reduce(Post.none) do |posts, literal|
posts.or(tag_literal_relation(locale, literal[:name], negative: literal[:negative],
reference: literal[:reference]))
posts.or(tag_literal_relation(locale, literal[:name], negative: literal[:negative]))
end
else
literals.reduce(Post.all) do |posts, literal|
ids = tagged_post_ids_for(locale, literal[:name], reference: literal[:reference])
ids = tagged_post_ids_for(locale, literal[:name])
if literal[:negative]
posts.where.not(id: ids)
else
@@ -488,8 +485,8 @@ class PostsController < ApplicationController
end
end
def tag_literal_relation locale, name, negative:, reference: nil
ids = tagged_post_ids_for(locale, name, reference:)
def tag_literal_relation locale, name, negative:
ids = tagged_post_ids_for(locale, name)
if negative
Post.where.not(id: ids)
else
@@ -497,8 +494,8 @@ class PostsController < ApplicationController
end
end
def tagged_post_ids_for locale, name, reference: nil
internal, external = PostTagSearch.references(locale, { name:, reference: })
def tagged_post_ids_for locale, name
internal, external = PostTagSearch.references(locale, { name: })
Post.where(id: PostTag.where(tag_id: Array(internal)).select(:post_id))
.or(Post.where(id: PostExternalTag.where(external_tag_id: Array(external))
.select(:post_id))).select(:id)
+54
ファイルの表示
@@ -0,0 +1,54 @@
class StaticMirrorsController < ApplicationController
before_action :development_only
wrap_parameters false
def show
render json: { resource_limit: StaticMirror::RESOURCE_LIMIT,
name_limit: StaticMirror::NAME_LIMIT, max_bytes: StaticMirror::MAX_BYTES,
max_seconds: StaticMirrorJob::MAX_SECONDS, concurrency: 1,
cooldown_seconds: StaticMirrorJob::COOLDOWN,
expiry_seconds: StaticMirrorJob::EXPIRY.to_i, exclusions: StaticMirror::EXCLUSIONS }
end
def create
# No date input is accepted: this endpoint cannot promise historical restoration.
payload = request.request_parameters.except('authenticity_token')
.merge(request.query_parameters)
return head :bad_request unless (payload.keys - ['locale']).empty?
locale = resolve_locale!
session[:static_mirror_owner] ||= SecureRandom.hex(32)
job = StaticMirrorJob.start(owner: session[:static_mirror_owner], locale: locale.code)
render json: { job: }, status: :accepted
rescue StaticMirrorJob::Busy
response.headers['Retry-After'] = StaticMirrorJob::COOLDOWN.to_s
head :too_many_requests
rescue ActiveJob::EnqueueError
render json: { error: 'generation_failed' }, status: :service_unavailable
end
def status
value = StaticMirrorJob.status(params[:id], session[:static_mirror_owner])
return head :not_found unless value
render json: value
rescue StaticMirrorJob::Busy
head :service_unavailable
end
def download
bytes = StaticMirrorJob.download(params[:id], session[:static_mirror_owner])
return head :not_found unless bytes
send_data bytes, filename: 'static-mirror.zip', type: 'application/zip',
disposition: 'attachment'
rescue StaticMirrorJob::Busy
head :service_unavailable
end
private
def development_only
head :not_found unless Rails.env.development?
end
end
+166
ファイルの表示
@@ -0,0 +1,166 @@
require 'fileutils'
require 'timeout'
# Development PoC: one filesystem slot and Rails' in-process async executor.
# A process restart loses queued work; stale state fails rather than reporting success.
class StaticMirrorJob < ApplicationJob
self.queue_adapter = :async
EXPIRY = 10.minutes
COOLDOWN = 60
MAX_SECONDS = 60
ACTIVE = ['queued', 'running'].freeze
class Busy < StandardError; end
def self.directory = Rails.root.join('tmp/static_mirror')
def self.synchronise(blocking: false)
FileUtils.mkdir_p(directory, mode: 0o700)
File.open(directory.join('.registry.lock'), File::RDWR | File::CREAT, 0o600) do |lock|
mode = blocking ? File::LOCK_EX : File::LOCK_EX | File::LOCK_NB
raise Busy unless lock.flock(mode)
yield
end
end
def self.state
JSON.parse(directory.join('state.json').read)
rescue Errno::ENOENT
nil
end
def self.write(value)
path = directory.join('state.tmp')
File.open(path, 'w', 0o600) { _1.write(JSON.generate(value)) }
File.rename(path, directory.join('state.json'))
ensure
File.unlink(path) if path&.exist?
end
def self.expire_stalled(value)
return unless value && ACTIVE.include?(value['status'])
return unless Time.current.to_f - value.fetch('created_at') > MAX_SECONDS + 15
value.merge!('status' => 'failed', 'error' => 'worker_stopped')
write(value)
end
def self.start(owner:, locale:)
id = synchronise do
previous = state
expire_stalled(previous)
raise Busy if previous && ACTIVE.include?(previous['status'])
if previous && Time.current.to_f - previous.fetch('created_at') < COOLDOWN
raise Busy
end
File.open(directory.join('.generation.lock'), File::RDWR | File::CREAT, 0o600) do |lock|
raise Busy unless lock.flock(File::LOCK_EX | File::LOCK_NB)
FileUtils.rm_f(directory.join('archive.zip'))
FileUtils.rm_f(directory.join('archive.part'))
identifier = SecureRandom.hex(24)
write({ 'id' => identifier, 'owner' => owner, 'locale' => locale,
'created_at' => Time.current.to_f, 'expiry_at' => (Time.current + EXPIRY).to_f,
'status' => 'queued', 'processed' => 0, 'total' => nil, 'data_at' => nil })
identifier
end
end
perform_later(id)
id
rescue ActiveJob::EnqueueError
update(id) { _1.merge!('status' => 'failed', 'error' => 'generation_failed') } if id
raise
end
def self.authorised_state(id, owner)
value = state
return unless owner.present? && value && value['id'] == id && value['owner'] == owner
return unless value['expiry_at'] > Time.current.to_f
expire_stalled(value)
if value['status'] == 'ready'
permitted = StaticMirror.authorised_artifact?(value)
return unless permitted && directory.join('archive.zip').file?
end
value
end
def self.status(id, owner)
synchronise do
authorised_state(id, owner)&.slice('status', 'processed', 'total', 'data_at', 'error')
end
end
def self.download(id, owner)
synchronise do
value = authorised_state(id, owner)
return unless value && value['status'] == 'ready'
bytes = directory.join('archive.zip').binread
# Revalidate after reading; send_data avoids delayed sendfile reuse of a stale path.
permitted = StaticMirror.authorised_artifact?(value)
permitted ? bytes : nil
end
end
def self.update(id)
# Worker updates may wait for a short status/download read; request intake never waits.
synchronise(blocking: true) do
value = state
return unless value && value['id'] == id
# Expiry is terminal even when a delayed worker still owns the generation lock.
return if value['status'] == 'failed'
yield value
write(value)
value
end
end
def perform(id)
File.open(self.class.directory.join('.generation.lock'), File::RDWR) do |lock|
# Intake briefly owns this lock. A queued worker must wait, never disappear.
lock.flock(File::LOCK_EX)
value = self.class.state
return unless value && value['id'] == id && value['status'] == 'queued'
generate(id, value)
end
end
private
def generate(id, value)
return unless self.class.update(id) { _1['status'] = 'running' }
remaining = MAX_SECONDS - (Time.current.to_f - value.fetch('created_at'))
raise Timeout::Error unless remaining.positive?
Timeout.timeout(remaining) do
locale = Locale.find_by!(code: value.fetch('locale'), deprecated_at: nil)
mirror = StaticMirror.new(locale:)
bytes = mirror.export do |processed, total|
self.class.update(id) do |metadata|
metadata.merge!('processed' => processed, 'total' => total, 'data_at' => mirror.data_at)
end
end
part = self.class.directory.join('archive.part')
File.open(part, 'wb', 0o600) { _1.write(bytes) }
File.rename(part, self.class.directory.join('archive.zip'))
self.class.update(id) do |metadata|
metadata.merge!(mirror.authority).merge!('status' => 'ready', 'data_at' => mirror.data_at)
end
end
rescue StandardError => error
FileUtils.rm_f(self.class.directory.join('archive.part'))
FileUtils.rm_f(self.class.directory.join('archive.zip'))
code = case error
when StaticMirror::TooLarge then 'too_large'
when Timeout::Error then 'generation_timeout'
when HistoryAccess::Forbidden then 'publication_changed'
else 'generation_failed'
end
self.class.update(id) { _1.merge!('status' => 'failed', 'error' => code) }
Rails.logger.warn("static_mirror_failure: #{ error.class }")
end
end
+19 -6
ファイルの表示
@@ -1,8 +1,11 @@
class ExternalTag < ApplicationRecord
before_create { SharedResourceIdAllocator.assign!(self) }
# Must run before dependent association callbacks.
before_destroy :prevent_destroy
enum :platform, nico: 'nico'
validates :platform, presence: true, inclusion: { in: ExternalTag.platforms.keys }
validate :immutable_identity, on: :update
has_many :post_external_tags, dependent: :delete_all
has_many :posts, through: :post_external_tags
@@ -16,11 +19,21 @@ class ExternalTag < ApplicationRecord
has_many :linked_tags, through: :nico_tag_relations, source: :tag
def snapshot_linked_tag_names language_code = 'ja'
linked_tags
.joins(:tag_names)
.where(tag_names: { language_code:, primary_flg: true })
.order('tag_names.name')
.pluck('tag_names.name')
def snapshot_linked_tag_ids = linked_tags.distinct.order(:id).pluck(:id)
def delete
raise ActiveRecord::ReadOnlyRecord, '外部タグは削除できません.'
end
private
def immutable_identity
errors.add(:platform, 'は変更できません.') if will_save_change_to_platform?
errors.add(:name, 'は変更できません.') if will_save_change_to_name?
end
def prevent_destroy
errors.add(:base, '外部タグは削除できません.')
throw(:abort)
end
end
+34 -6
ファイルの表示
@@ -4,6 +4,7 @@ require 'digest'
class HistoryAccess < ActiveSupport::CurrentAttributes
attribute :user
class Forbidden < StandardError; end
ARTIFACT_POLICY = 'counts-private-predicates-4'.freeze
KEYS = { 'post_versions' => ['post_id'], 'tag_versions' => ['tag_id'],
'nico_tag_versions' => ['tag_id'],
@@ -18,10 +19,12 @@ class HistoryAccess < ActiveSupport::CurrentAttributes
end
def self.authority
{ 'policy_revision' => revision, 'admin' => admin?, 'actor_id' => user&.id }
{ 'artifact_policy' => ARTIFACT_POLICY, 'policy_revision' => revision,
'admin' => admin?, 'actor_id' => user&.id }
end
def self.authorised_artifact?(metadata)
return false unless metadata['artifact_policy'] == ARTIFACT_POLICY
return false unless metadata['policy_revision'].is_a?(String)
if metadata['admin']
admin? && user.id == metadata['actor_id']
@@ -56,20 +59,41 @@ class HistoryAccess < ActiveSupport::CurrentAttributes
end
def self.json_reference(table, key, document)
"NOT EXISTS (SELECT 1 FROM JSON_TABLE(COALESCE(#{ document }, JSON_ARRAY()), " +
# A scalar lookup avoids MySQL's semijoin rewrite losing the JSON row correlation.
"(SELECT 1 FROM JSON_TABLE(COALESCE(#{ document }, JSON_ARRAY()), " +
"'$[*]' COLUMNS (reference_id BIGINT PATH '$' NULL ON EMPTY)) reference_entry " +
"JOIN (#{ discarded_keys(table, key) }) blocked " +
"ON reference_entry.reference_id = blocked.#{ key })"
"ON reference_entry.reference_id = blocked.#{ key } LIMIT 1) IS NULL"
end
def self.list_reference(table, key, document)
# Preserve FIND_IN_SET's exact token semantics, including empty tokens.
array = "CONCAT('[', REPLACE(JSON_QUOTE(REPLACE(COALESCE(#{ document }, ''), " +
"' ', ',')), ',', '\",\"'), ']')"
"NOT EXISTS (SELECT 1 FROM JSON_TABLE(#{ array }, '$[*]' " +
"(SELECT 1 FROM JSON_TABLE(#{ array }, '$[*]' " +
"COLUMNS (reference_id VARCHAR(255) PATH '$')) reference_entry " +
"JOIN (#{ discarded_keys(table, key) }) blocked " +
"ON reference_entry.reference_id = CAST(blocked.#{ key } AS CHAR))"
"ON reference_entry.reference_id = CAST(blocked.#{ key } AS CHAR) LIMIT 1) IS NULL"
end
def self.snapshot_reference(model, row, external: false)
key = external ? 'external_tag_id' : 'tag_id'
table = model.table_name
source = "JSON_TABLE(COALESCE(#{ row }.tags_json, JSON_ARRAY()), '$[*]' COLUMNS (" +
"tag_id BIGINT PATH '$.#{ key }', version_no INT PATH '$.version_no')) tag_reference"
match = 'reference_version.tag_id = tag_reference.tag_id'
if external
match += " AND reference_version.created_at <= #{ row }.created_at"
latest = "AND NOT EXISTS (SELECT 1 FROM #{ table } later_reference " +
'WHERE later_reference.tag_id = reference_version.tag_id ' +
"AND later_reference.created_at <= #{ row }.created_at " +
'AND later_reference.version_no > reference_version.version_no)'
else
match += ' AND reference_version.version_no = tag_reference.version_no'
latest = ''
end
"(SELECT 1 FROM #{ source } JOIN #{ table } reference_version ON #{ match } " +
"WHERE NOT (#{ sql(model, 'reference_version') }) #{ latest } LIMIT 1) IS NULL"
end
def self.reference_sql(table, row)
@@ -78,7 +102,11 @@ class HistoryAccess < ActiveSupport::CurrentAttributes
[['tag_versions', 'tag_id', 'tag_id'],
['nico_tag_versions', 'tag_id', 'external_tag_id']].map do |target, key, field|
json_reference(target, key, "JSON_EXTRACT(#{ row }.tags_json, '$[*].#{ field }')")
end + [list_reference('post_versions', 'post_id', "#{ row }.parent_post_ids")]
end + [list_reference('post_versions', 'post_id', "#{ row }.parent_post_ids"),
snapshot_reference(TagVersion, row),
snapshot_reference(NicoTagVersion, row, external: true)]
when 'nico_tag_versions'
[list_reference('tag_versions', 'tag_id', "#{ row }.linked_tag_ids")]
when 'tag_versions'
[list_reference('tag_versions', 'tag_id', "#{ row }.parent_tag_ids")]
when 'tag_description_versions'
+6 -1
ファイルの表示
@@ -3,5 +3,10 @@ class NicoTagVersion < ApplicationRecord
belongs_to :external_tag, foreign_key: :tag_id, inverse_of: :nico_tag_versions
validates :name, presence: true
validates :event_type, inclusion: { in: ['create', 'update'] }
def name
tag = external_tag
"#{ tag.platform }:#{ tag.name }" if tag
end
end
+4 -1
ファイルの表示
@@ -139,6 +139,8 @@ class CountsVersionRepr
result = {}
table = model.table_name
comparison = @inclusive ? '<=' : '<'
name = model == NicoTagVersion ?
CountsResourceDefinition.new('nico_tags').expression('name', table) : 'name'
requests.each_slice(100) do |batch|
queries = batch.map do |id, point|
scope = HistoryAccess.visible(model.where(tag_id: id))
@@ -148,7 +150,8 @@ class CountsVersionRepr
"AND newer.version_no > #{ table }.version_no)", point)
quoted = model.connection.quote(point.iso8601(6))
names = model == TagVersion ? 'names' : 'JSON_OBJECT() AS names'
scope.select("tag_id, name, #{ names }, #{ quoted } AS reference_point").to_sql
scope.select("tag_id, #{ name } AS name, #{ names }, " +
"#{ quoted } AS reference_point").to_sql
end
next if queries.empty?
+85 -3
ファイルの表示
@@ -1,6 +1,41 @@
class CountsAxes
MAX_AXES = 8
MASKED_TEXT = '********'.freeze
MASKED_VALUE = '__counts_private__'.freeze
PUBLIC_FIELDS = ['resource_id', 'version_no', 'event_type', 'user',
'created_at'].freeze
class TooManyAxes < ArgumentError; end
class PrivateCondition < ArgumentError; end
GRAINS = ['year', 'quarter', 'month', 'week', 'day', 'hour'].freeze
def self.private_field?(field)
!PUBLIC_FIELDS.include?(field) && field != 'user.id'
end
def self.visible_ids_sql(definition)
"SELECT counts_visible.id FROM #{ definition.table } counts_visible WHERE " +
HistoryAccess.sql(definition.model, 'counts_visible')
end
def self.visibility_sql(definition, field)
return 'TRUE' if HistoryAccess.admin?
source = visible_ids_sql(definition)
visible = "v.id IN (#{ source })"
return visible unless field == 'changed_field'
"(#{ visible }) AND (previous.id IS NULL OR previous.id IN (#{ source }))"
end
def self.masked_sql(expression, definition, field)
return expression if HistoryAccess.admin? || !private_field?(field)
visible = visibility_sql(definition, field)
quoted = ApplicationRecord.connection.quote(MASKED_VALUE)
"CASE WHEN #{ visible } THEN #{ expression } ELSE #{ quoted } END"
end
def self.time_expression(expression, grain, timezone)
raise ArgumentError unless GRAINS.include?(grain)
@@ -36,13 +71,14 @@ class CountsAxes
@rows = parse(population.params.fetch('rows', ''))
@columns = parse(population.params.fetch('cols', ''))
@all = rows + columns
raise ArgumentError if all.length > 8 || all.map { _1['field'] }.uniq.length != all.length
raise TooManyAxes if all.length > MAX_AXES
raise ArgumentError if all.map { _1['field'] }.uniq.length != all.length
end
def expressions
@expressions ||= all.map do |axis|
field = axis.fetch('field')
case field
expression = case field
when 'tags'
"CASE WHEN tag_entry.tag_id IS NOT NULL " +
"THEN CONCAT('internal:', tag_entry.tag_id) " +
@@ -53,14 +89,53 @@ class CountsAxes
@definition.fields[field] == :datetime ? self.class.time_expression(
expression, axis.fetch('grain', 'month'), @population.timezone) : expression
end
if !HistoryAccess.admin? && self.class.private_field?(field)
expression = self.class.masked_sql(
"CAST((#{ expression }) AS CHAR)", @definition, field)
end
expression
end
end
def grouped_sql(scope, indexes = all.each_index.to_a)
keys = indexes.map { "a#{ _1 }" }
selected = indexes.map { "#{ expressions[_1] } AS a#{ _1 }" }
identities = ['v.id']
if rows.any? { _1['field'] == 'tags' }
identities << self.class.masked_sql(
"COALESCE(CONCAT('internal:', tag_entry.tag_id), " +
"CONCAT('external:', tag_entry.external_tag_id))", @definition, 'tags')
if rows.any? { _1['field'] == 'changed_field' }
identities << self.class.masked_sql('changed.field_key', @definition, 'changed_field')
end
end
members = identities.each_index.map { "counts_member_#{ _1 }" }
selected += identities.each_with_index.map { |expression, index|
"#{ expression } AS #{ members[index] }"
}
# Evaluate correlated visibility subqueries before grouping. LIMIT prevents
# derived-table merging, so ONLY_FULL_GROUP_BY sees ordinary projected columns.
projection = scope.except(:select, :order).select(Arel.sql(selected.join(', ')))
.limit(18_446_744_073_709_551_615).to_sql
fields = keys + ["COUNT(DISTINCT #{ members.join(', ') }) AS count"]
grouping = keys.empty? ? '' : " GROUP BY #{ keys.join(', ') }"
"SELECT #{ fields.join(', ') } FROM (#{ projection }) counts_values#{ grouping }"
end
def joins(fields = all.map { _1['field'] })
result = ''
if fields.include?('tags')
document = 'CAST(v.tags_json AS JSON)'
unless HistoryAccess.admin?
visible = self.class.visible_ids_sql(@definition)
result += " LEFT JOIN (#{ visible }) counts_tags_visible " +
'ON counts_tags_visible.id = v.id '
document = "CASE WHEN counts_tags_visible.id IS NOT NULL " +
"THEN #{ document } ELSE JSON_ARRAY(" +
"JSON_OBJECT('tag_id', 0)) END"
end
result += <<~SQL.squish
JOIN JSON_TABLE(CAST(v.tags_json AS JSON), '$[*]' COLUMNS (
JOIN JSON_TABLE(#{ document }, '$[*]' COLUMNS (
tag_id BIGINT PATH '$.tag_id' NULL ON EMPTY,
external_tag_id BIGINT PATH '$.external_tag_id' NULL ON EMPTY)) tag_entry
@@ -73,6 +148,13 @@ class CountsAxes
condition = @definition.changes.map do |field|
"(changed.field_key = '#{ field }' AND #{ @population.changed_sql(field) })"
end.join(' OR ')
unless HistoryAccess.admin?
visible = self.class.visibility_sql(@definition, 'changed_field')
marker = ApplicationRecord.connection.quote(MASKED_VALUE)
fields += " UNION ALL SELECT #{ marker } AS field_key"
condition = "((#{ visible }) AND (#{ condition })) OR " +
"(NOT (#{ visible }) AND changed.field_key = #{ marker })"
end
result += " JOIN (#{ fields }) changed ON (#{ condition })"
end
result
+7 -6
ファイルの表示
@@ -28,30 +28,31 @@ class CountsCandidates
[external, 'external', 'NULL', 'JSON_OBJECT()']]
sources = sources.map do |population, kind, category, names|
scope = population.scope.except(:select)
name = population.definition.expression('name')
if @params['query'].present?
pattern = "%#{ ApplicationRecord.sanitize_sql_like(@params['query']) }%"
scope = scope.where("(v.name LIKE ? OR CAST(v.tag_id AS CHAR) = ? OR EXISTS (" +
scope = scope.where("(#{ name } LIKE ? OR CAST(v.tag_id AS CHAR) = ? OR EXISTS (" +
"SELECT 1 FROM JSON_TABLE(COALESCE(JSON_EXTRACT(#{ names }, '$**.name'), " +
"JSON_ARRAY()), '$[*]' COLUMNS (name TEXT PATH '$')) candidate_name " +
'WHERE candidate_name.name LIKE ?))', pattern, @params['query'], pattern)
end
scope.select(Arel.sql(
"CONCAT('#{ kind }:', v.tag_id) AS value, v.tag_id AS id, " +
"v.name AS name, #{ names } AS names, #{ category } AS category, '#{ kind }' AS kind"))
"#{ name } AS name, #{ names } AS names, " +
"#{ category } AS category, '#{ kind }' AS kind"))
.to_sql
end
clauses = []
binds = []
if @params['category'].present?
clauses << 'category = ?'
binds << @params['category']
clauses << ApplicationRecord.sanitize_sql_array(['category = ?', @params['category']])
end
where = clauses.empty? ? '' : "WHERE #{ clauses.join(' AND ') }"
sql = <<~SQL
SELECT /*+ MAX_EXECUTION_TIME(10000) */ * FROM (#{ sources.join(' UNION ALL ') }) choices
#{ where } ORDER BY kind, id LIMIT #{ LIMIT + 1 } OFFSET #{ (@page - 1) * LIMIT }
SQL
rows = connection.select_all(ApplicationRecord.sanitize_sql_array([sql, *binds])).to_a
# Sources already contain quoted search terms. Do not reinterpret their % or ? as binds.
rows = connection.select_all(sql).to_a
rows.each { _1['names'] = JSON.parse(_1['names']) if _1['names'].is_a?(String) }
{ rows: rows.first(LIMIT), page: @page, more: rows.length > LIMIT,
execution_at: internal.at.iso8601(6) }
+51 -21
ファイルの表示
@@ -82,6 +82,34 @@ class CountsCondition
def apply(scope, entry)
raise ArgumentError unless entry.is_a?(Hash)
if entry['operator'] == 'axis_eq'
raise ArgumentError unless (entry.keys - ['field', 'operator', 'grain', 'value']).empty?
value = entry['value']
unless (value.nil? || value.is_a?(String) || value.is_a?(Numeric) ||
[true, false].include?(value))
raise ArgumentError
end
raise ArgumentError if value.is_a?(String) && value.bytesize > 8192
if (!HistoryAccess.admin? && CountsAxes.private_field?(entry['field']) &&
(value == CountsAxes::MASKED_VALUE ||
(value == CountsAxes::MASKED_TEXT &&
(@definition.fields[entry['field']] != :string ||
['tags', 'changed_field'].include?(entry['field'])))))
field = entry.fetch('field')
raise ArgumentError unless @definition.axes(@population.history).include?(field)
if entry.key?('grain')
unless (@definition.fields[field] == :datetime &&
CountsAxes::GRAINS.include?(entry['grain']))
raise ArgumentError
end
end
return scope.none
end
scope = @population.condition_scope(scope, entry.fetch('field'))
return @population.axis_condition(scope, entry)
end
allowed = ['field', 'operator', 'value', 'end', 'blank', 'input', 'values']
raise ArgumentError if (entry.keys - allowed).any?
@@ -101,6 +129,13 @@ class CountsCondition
source, expression = source_for(field[:key])
type = field[:type]
# Preserve typed input compatibility; display markers are never real typed values.
if (!HistoryAccess.admin? && CountsAxes.private_field?(field[:key]) &&
entry['value'] == CountsAxes::MASKED_TEXT && type != :string && !entry['blank'])
return scope.none unless entry['operator'] == 'ne'
return @population.condition_scope(scope, field[:key])
end
blank = blank_sql(expression, type)
if entry['blank']
sql = if source && field[:collection]
@@ -109,6 +144,7 @@ class CountsCondition
"NOT EXISTS (SELECT 1 FROM #{ source } WHERE NOT (#{ blank }))"
else blank
end
scope = @population.condition_scope(scope, field[:key])
return scope.where(Arel.sql(sql))
end
if ['in', 'not_in', 'any', 'all', 'none'].include?(entry['operator'])
@@ -121,6 +157,7 @@ class CountsCondition
validate_enum!(field[:key], parsed.first) if type == :enum
operator = entry['operator']
if type == :interval
scope = @population.condition_scope(scope, field[:key])
return interval(scope, source, expression, parsed)
end
clauses = []
@@ -148,28 +185,10 @@ class CountsCondition
elsif operator == 'ne'
predicate = "(NOT (#{ predicate }) OR #{ blank })"
end
scope = @population.condition_scope(scope, field[:key])
scope.where(predicate, *binds)
end
def tag_name_match(value)
source = post_tag_source
bound = @population.history ? 'v.created_at' : @connection.quote(point)
source += ' LEFT JOIN nico_tag_versions external_snapshot ON external_snapshot.tag_id = ' +
scalar('tag_snapshot.item', 'external_tag_id', :integer) +
" AND external_snapshot.created_at <= #{ bound } AND " +
@population.access_sql(NicoTagVersion, 'external_snapshot') +
' AND NOT EXISTS (SELECT 1 FROM nico_tag_versions newer_external ' +
'WHERE newer_external.tag_id = external_snapshot.tag_id ' +
"AND newer_external.created_at <= #{ bound } " +
'AND newer_external.version_no > external_snapshot.version_no)'
document = "JSON_MERGE_PRESERVE(JSON_ARRAY(#{ scalar('tag_snapshot.item', 'name') }, " +
"referenced_tag.name, CONCAT('nico:', external_snapshot.name)), " +
"COALESCE(JSON_EXTRACT(referenced_tag.names, '$**.name'), JSON_ARRAY()))"
source += ' JOIN ' + json_table(document, '$[*]', 'search_name')
@definition.model.sanitize_sql_array([
"EXISTS (SELECT 1 FROM #{ source } WHERE JSON_UNQUOTE(search_name.item) = ?)", value])
end
private
def set_condition(scope, entry, field)
@@ -177,6 +196,13 @@ class CountsCondition
values = entry['values']
raise ArgumentError unless values.is_a?(Array) && values.length <= 100
if (!HistoryAccess.admin? && CountsAxes.private_field?(field[:key]) &&
values.include?(CountsAxes::MASKED_TEXT) &&
(field[:type] != :string || field[:set_type] == 'tag_reference'))
return scope.none if entry['operator'] == 'all'
values = values - [CountsAxes::MASKED_TEXT]
end
raise ArgumentError if [entry['value'], entry['end']].any? { !_1.nil? && _1 != '' }
parsed = values.map do |value|
@@ -200,7 +226,10 @@ class CountsCondition
operator = entry['operator']
if parsed.empty?
return ['in', 'any'].include?(operator) ? scope.none : scope
return scope.none if ['in', 'any'].include?(operator)
return scope if entry['values'].empty?
return @population.condition_scope(scope, field[:key])
end
source, expression = source_for(field[:key])
if field[:set_type] == 'tag_reference'
@@ -221,6 +250,7 @@ class CountsCondition
predicate = source ? "NOT #{ predicate }" :
"(NOT #{ predicate } OR #{ blank_sql(expression, field[:type]) })"
end
scope = @population.condition_scope(scope, field[:key])
scope.where(predicate, *parsed)
end
@@ -349,7 +379,7 @@ class CountsCondition
return @definition.alias_source(root) if root.start_with?('aliases_')
return [nil, @definition.name_expression(root)]
end
return [nil, "(#{ @population.latest_sql })"] if root == 'latest'
return [nil, "(#{ @population.current_latest_sql })"] if root == 'latest'
if root == 'user' && parts.any?
expression = parts.first == 'id' ? 'v.created_by_user_id' :
'(SELECT actor.name FROM users actor WHERE actor.id = v.created_by_user_id)'
+1 -1
ファイルの表示
@@ -20,7 +20,7 @@ class CountsDefaults
details = details.transform_values do |value|
value.split(' ').flat_map do |token|
if resource == 'tags' && ['name', 'parents.name'].include?(token)
token == 'name' ? name : ["parents.#{ name }", "parents.#{ aliases }"]
token == 'name' ? name : "parents.#{ name }"
elsif resource == 'tags' && token == 'aliases'
aliases
elsif resource == 'tags' && token.start_with?('aliases.')
+47
ファイルの表示
@@ -13,6 +13,53 @@ class CountsExportDownload
class TimedOut < StandardError; end
class StorageFull < StandardError; end
# Rack body deliberately has no to_path: sendfile middleware must not bypass ACK.
class ResponseBody
def initialize body, &state
@body = body
@closed = false
@state = state
end
def each &block
return enum_for(:each) unless block
return if @closed
transmit(&block)
end
def close
return if @closed
@closed = true
@body.close if @body.respond_to?(:close)
end
private
def transmit
first = true
@body.each do |chunk|
raise IOError, 'Download no longer authorised' unless @state.call('active')
yield chunk
if first
# Rack accepted a chunk; browser receipt and saving are not guaranteed.
raise IOError, 'Download acknowledgement failed' unless @state.call('started')
first = false
end
end
@state.call('failed') if first
rescue StandardError => error
Rails.logger.warn("counts_download_stream_failure: #{ error.class }")
@state.call('failed')
raise
ensure
close
end
end
def self.directory
Rails.root.join(ENV.fetch('COUNTS_EXPORT_DIRECTORY', 'tmp/counts_exports'))
end
+83 -3
ファイルの表示
@@ -45,6 +45,20 @@ class CountsExportJobs
write(state)
end
def self.failure_code(error)
case error
when Cancelled then 'cancelled'
when CountsAxes::TooManyAxes then 'too_many_axes'
when CountsAxes::PrivateCondition then 'private_condition'
when CountsTableQuery::TooManyColumns then 'too_many_columns'
when ActiveRecord::StatementTimeout then 'counts_query_timeout'
when CountsExportDownload::TimedOut then 'generation_timeout'
when CountsExportDownload::TooLarge then 'export_too_large'
when CountsExportDownload::StorageFull, Errno::ENOSPC then 'export_storage_full'
else 'generation_failed'
end
end
def self.directory = CountsExportDownload.directory.join('jobs')
def self.verifier = Rails.application.message_verifier('counts_export_job')
@@ -176,6 +190,12 @@ class CountsExportJobs
return unless state && state['owner'] == owner && state['expires_at'] > Time.current.to_f
expire_stalled(state)
if (state['status'] == 'ready' &&
['pending', 'streaming'].include?(state['download_status']) &&
state.fetch('download_expires_at', 0) <= Time.current.to_f)
state['download_status'] = 'timeout'
write(state)
end
if cancel && ['queued', 'running', 'ready'].include?(state['status'])
state['status'] = state['status'] == 'running' ? 'cancelling' : 'cancelled'
@@ -183,11 +203,71 @@ class CountsExportJobs
state.delete('token')
write(state)
end
state.slice('status', 'phase', 'processed', 'total', 'format', 'token', 'error',
'updated_at', 'heartbeat_at')
result = state.slice('status', 'phase', 'processed', 'total', 'format', 'token', 'error',
'updated_at', 'heartbeat_at', 'download_attempt', 'download_status')
result['max_axes'] = CountsAxes::MAX_AXES if state['error'] == 'too_many_axes'
result['max_columns'] = CountsTableQuery::MAX_COLUMNS if state['error'] == 'too_many_columns'
result
end
end
def self.download_attempt(token, owner, file_token: nil, attempt: nil, status: nil)
id = authorised_id(token, owner)
return unless id
synchronise do
state = read(id)
return unless (state && state['owner'] == owner && state['status'] == 'ready' &&
state['expires_at'] > Time.current.to_f && HistoryAccess.authorised_artifact?(state))
if attempt
return unless state['download_attempt'] == attempt && state['token'] == file_token
return unless state['download_status'] == 'started' ||
state.fetch('download_expires_at', 0) > Time.current.to_f
transitions = { 'streaming' => ['pending'], 'started' => ['streaming'],
'failed' => ['pending', 'streaming', 'started'],
'unavailable' => ['pending', 'streaming'] }
if status == 'active'
return state.slice('download_attempt', 'download_status', 'token') if
['streaming', 'started'].include?(state['download_status'])
return
end
if status
return unless transitions.fetch(status, []).include?(state['download_status'])
state['download_status'] = status
else
return unless state['download_status'] == 'pending'
end
else
# Repeated acceptance POSTs do not invalidate an in-flight GET or issue duplicates.
if (state['download_status'] == 'started' ||
(['pending', 'streaming'].include?(state['download_status']) &&
state.fetch('download_expires_at', 0) > Time.current.to_f))
return state.slice('download_attempt', 'download_status', 'token')
end
state['download_attempt'] = SecureRandom.hex(24)
state['download_status'] = 'pending'
state['download_expires_at'] = Time.current.to_f + 45
end
write(state)
state.slice('download_attempt', 'download_status', 'token')
end
end
# GET already verified authority. During streaming, observe cancellation without
# repeating the six-table policy revision aggregate for every 64KiB chunk.
def self.stream_active?(token, owner, file_token:, attempt:)
id = authorised_id(token, owner)
state = id && read(id)
state && state['owner'] == owner && state['status'] == 'ready' &&
state['expires_at'] > Time.current.to_f && state['token'] == file_token &&
state['download_attempt'] == attempt &&
(state['download_status'] == 'started' ||
(state['download_status'] == 'streaming' &&
state.fetch('download_expires_at', 0) > Time.current.to_f))
end
def self.checkpoint(id, processed = nil, total = nil, phase: nil)
synchronise do
state = read(id)
@@ -376,7 +456,7 @@ class CountsExportJobs
if current
unless current['status'] == 'failed'
current['status'] = error.is_a?(Cancelled) ? 'cancelled' : 'failed'
current['error'] = error.class.name
current['error'] = failure_code(error)
end
current['updated_at'] = Time.current.to_f
write(current)
+1 -1
ファイルの表示
@@ -157,7 +157,7 @@ class CountsExportProjection
def csv_display(value, key)
if key.start_with?('aliases_') && value.is_a?(Array)
return value.map { fetch(_1, 'name') }.compact.join("\n")
return value.map { fetch(_1, 'name') }.compact.join(' ')
end
return @labels.fetch('bot', '') if key == 'created_by_user' && value.nil?
return @labels.fetch("event_type:#{ value }", value) if key == 'event_type'
+8 -1
ファイルの表示
@@ -6,6 +6,13 @@ class CountsExporter
'summary_columns', 'nested_json', 'child_sort', 'labels'].freeze
attr_reader :format, :resource
def self.with_read_snapshot(&block)
# A caller-owned transaction already owns the read view (including fixtures).
# MySQL cannot change isolation inside it. Standalone workers always start RR.
options = ApplicationRecord.connection.transaction_open? ? {} : { isolation: :repeatable_read }
ApplicationRecord.transaction(**options, &block)
end
def initialize(params, language_code:)
params = CountsResourceDefinition.normalise_params(params)
if params.key?('definition')
@@ -52,7 +59,7 @@ class CountsExporter
return @table_export.body(progress:, phase:) if @table_export
Enumerator.new do |output|
ApplicationRecord.transaction(isolation: :repeatable_read) do
self.class.with_read_snapshot do
processed = 0
first_row = true
started = false
+1
ファイルの表示
@@ -28,6 +28,7 @@ class CountsPivotLayout
result = []
values.each_with_index do |tuple, index|
result << { values: tuple, kind: 'data' }
# A group continuing on the next page closes only on its final page.
following = values[index + 1] || next_values
depths.reverse_each do |depth|
prefix = tuple.first(depth)
+43 -18
ファイルの表示
@@ -1,5 +1,6 @@
# Re-aggregate the population, never add up already grouped cells (multi-value axes overlap).
# Re-aggregate distinct membership; tag row tuples alone use additive contributions.
class CountsPivotTotals
class TimedOut < ActiveRecord::StatementTimeout; end
def initialize(population, axes, row_headers, column_headers)
@population = population
@axes = axes
@@ -8,8 +9,12 @@ class CountsPivotTotals
end
def call
# MySQL's unlimited LIMIT prevents CTE merging: authorise once for all branches.
basis = @population.scope.except(:select, :order).select('v.*')
# Materialise full-cell membership once, including omitted multi-value axes.
membership = @population.scope.except(:select, :order)
joins = @axes.joins
membership = membership.joins(joins) unless joins.empty?
# Retain exactly the records contributing a full cell, including changed_field.
basis = membership.select('v.*').distinct
.limit(18_446_744_073_709_551_615).to_sql
queries = @rows.flat_map do |row_depth, headers|
@columns.filter_map do |column_depth, columns|
@@ -18,41 +23,59 @@ class CountsPivotTotals
indexes = (0...row_depth).to_a +
(0...column_depth).map { @axes.rows.length + _1 }
expressions = indexes.map { @axes.expressions[_1] }
scope = @population.scope.except(:select, :order, :where, :from).from('authorised v')
joins = @axes.joins(indexes.map { @axes.all[_1].fetch('field') })
scope = @population.scope.except(:select, :order, :where, :from).from('population_members v')
additive = @axes.rows.any? { _1['field'] == 'tags' }
retained = additive ? (0...@axes.rows.length).to_a | indexes : indexes
joins = @axes.joins(retained.map { @axes.all[_1].fetch('field') })
scope = scope.joins(joins) unless joins.empty?
predicates = []
if row_depth.positive?
tuples = headers.map { _1[:values] }.uniq
predicate = tuples.map { Array.new(row_depth) { |i|
"(#{ expressions[i] }) <=> ?"
"a#{ indexes[i] } <=> ?"
}.join(' AND ') }.map { "(#{ _1 })" }.join(' OR ')
scope = scope.where(predicate, *tuples.flatten)
predicates << @population.definition.model.sanitize_sql_array([
tuples.empty? ? 'FALSE' : "(#{ predicate })", *tuples.flatten])
end
if column_depth.positive?
tuples = columns.map { _1[:values] }.uniq
predicate = tuples.map { Array.new(column_depth) { |i|
"(#{ expressions[row_depth + i] }) <=> ?"
"a#{ indexes[row_depth + i] } <=> ?"
}.join(' AND ') }.map { "(#{ _1 })" }.join(' OR ')
scope = scope.where(predicate, *tuples.flatten)
predicates << @population.definition.model.sanitize_sql_array([
tuples.empty? ? 'FALSE' : "(#{ predicate })", *tuples.flatten])
end
grouped = expressions.each_with_index.map { |expression, index|
"#{ expression } AS v#{ index }"
# Filter projected coordinates, not repeated masking/time expressions.
# Each member still participates in the complete DISTINCT population.
selection = predicates.empty? ? '' : " WHERE #{ predicates.join(' AND ') }"
grouped = indexes.each_with_index.map { |key, index|
"a#{ key } AS v#{ index }"
}
# Fixed-width result coordinates, independent of NULL and of translated labels.
padding = (expressions.length...@axes.all.length).map { "NULL AS v#{ _1 }" }
fields = ["#{ row_depth } AS row_depth", "#{ column_depth } AS column_depth"] +
grouped + padding + ['COUNT(DISTINCT v.id) AS count']
scope = scope.group(*expressions.map { Arel.sql(_1) }) if expressions.any?
scope.optimizer_hints('MAX_EXECUTION_TIME(10000)')
.select(Arel.sql(fields.join(', '))).to_sql
grouped + padding + ['count']
if additive
# Sum full row tuples, not page cells; preserve DISTINCT inside each tuple.
inner = @axes.grouped_sql(scope, retained)
coordinates = indexes.map { "a#{ _1 }" }
projected = coordinates.each_with_index.map { |key, index| "#{ key } AS v#{ index }" }
output = ["#{ row_depth } AS row_depth", "#{ column_depth } AS column_depth"] +
projected + padding + ['SUM(count) AS count']
grouping = coordinates.empty? ? '' : " GROUP BY #{ coordinates.join(', ') }"
"SELECT #{ output.join(', ') } FROM (#{ inner }) leaf#{ selection }#{ grouping }"
else
inner = @axes.grouped_sql(scope, indexes)
"SELECT #{ fields.join(', ') } FROM (#{ inner }) grouped#{ selection }"
end
end
end
return [] if queries.empty?
sql = queries.map { "(#{ _1 })" }.join(' UNION ALL ')
@population.definition.model.connection.select_all(
"WITH authorised AS (#{ basis }) " +
"SELECT /*+ MAX_EXECUTION_TIME(10000) */ * FROM (#{ sql }) totals").map do |record|
statement = "WITH population_members AS (#{ basis }) " +
"SELECT /*+ #{ CountsTableQuery::SQL_HINTS.join(' ') } */ * FROM (#{ sql }) totals"
@population.definition.model.connection.select_all(statement).map do |record|
row_depth = record['row_depth'].to_i
column_depth = record['column_depth'].to_i
{ row: (0...row_depth).map { record["v#{ _1 }"] },
@@ -62,5 +85,7 @@ class CountsPivotTotals
row_depth, column_depth),
count: record['count'].to_i }
end
rescue ActiveRecord::StatementTimeout
raise TimedOut, 'Counts pivot totals exceeded the query time limit'
end
end
+90 -41
ファイルの表示
@@ -41,8 +41,7 @@ class CountsPopulation
raise ArgumentError
end
def latest_sql(row = 'v', cutoff = nil)
cutoff ||= history ? at : timestamp(params['at']) || at
def latest_at_sql(row, cutoff)
keys = definition.keys.map { "newer.#{ _1 } = #{ row }.#{ _1 }" }.join(' AND ')
bound = definition.model.connection.quote(cutoff)
<<~SQL.squish
@@ -52,16 +51,73 @@ class CountsPopulation
SQL
end
def current_latest_sql(row = 'v')
keys = definition.keys.map { "newer.#{ _1 } = #{ row }.#{ _1 }" }.join(' AND ')
<<~SQL.squish
NOT EXISTS (SELECT 1 FROM #{ definition.table } newer
WHERE #{ keys } AND newer.version_no > #{ row }.version_no)
SQL
end
def changed_sql(key)
column = definition.column(key)
"(previous.id IS NOT NULL AND NOT (v.#{ column } <=> previous.#{ column }))"
end
# Unrestricted predicates are used only for numerical totals, never record output.
# Aggregate sources are unrestricted; CountsAxes masks their output coordinates.
# Never serialise raw records from an unrestricted population.
def access_sql(model, row)
@authorisation_flg ? HistoryAccess.sql(model, row) : 'TRUE'
end
# Each predicate keeps its own visibility restriction, including OR branches.
def condition_scope(scope, field)
return scope if HistoryAccess.admin? || !CountsAxes.private_field?(field)
scope.where(Arel.sql(CountsAxes.visibility_sql(definition, field)))
end
def axis_condition(scope, entry)
raise CountsAxes::PrivateCondition if entry['value'] == CountsAxes::MASKED_VALUE
key = entry.fetch('field')
raise ArgumentError unless definition.axes(history).include?(key)
value = entry['value']
unless value.nil?
valid = case definition.fields[key]
when :integer
value.is_a?(Integer) || (value.is_a?(String) && value.match?(/\A-?[0-9]+\z/))
when :boolean then [true, false].include?(value)
else value.is_a?(String)
end
raise ArgumentError unless valid
end
if entry.key?('grain')
unless definition.fields[key] == :datetime && CountsAxes::GRAINS.include?(entry['grain'])
raise ArgumentError
end
end
if key == 'tags'
scope = scope.where(snapshot_membership(entry.fetch('value')))
elsif key == 'changed_field'
raise ArgumentError unless history && definition.changes.include?(entry['value'])
scope = scope.where(Arel.sql(changed_sql(entry['value'])))
elsif definition.fields[key] == :datetime && entry['grain']
scope = scope.where(definition.expression(key) => nil) if entry['value'].nil?
if entry['value']
expression = CountsAxes.time_expression(definition.expression(key),
entry['grain'], timezone)
scope = scope.where("#{ expression } = ?", entry['value'])
end
else
raise ArgumentError unless definition.axes(history).include?(key)
scope = scope.where("#{ definition.expression(key) } <=> ?", entry['value'])
end
scope
end
private
def make_population
@@ -69,7 +125,8 @@ class CountsPopulation
@scope = @scope.from("#{ table } v").select('v.*')
previous = ['rows', 'cols'].any? do |key|
QueryState.axes(params.fetch(key, '')).any? { _1['field'] == 'changed_field' }
end || QueryState.records(params, 'drill').any? { _1['field'] == 'changed_field' }
end || (QueryState.records(params, 'drill') + QueryState.records(params, 'conditions'))
.any? { _1['field'] == 'changed_field' }
if previous
keys = definition.keys.map { "previous.#{ _1 } = v.#{ _1 }" }.join(' AND ')
@scope = @scope.joins("LEFT JOIN #{ table } previous ON #{ keys } " +
@@ -82,7 +139,7 @@ class CountsPopulation
raise ArgumentError if point > at
@scope = @scope.where("v.#{ definition.time_column } <= ?", point)
.where(Arel.sql(latest_sql('v', point))).where.not('v.event_type' => 'discard')
.where(Arel.sql(latest_at_sql('v', point))).where.not('v.event_type' => 'discard')
return
end
first = timestamp(params['from'])
@@ -106,6 +163,14 @@ class CountsPopulation
end
apply_tags if params['tags'].present? || params['tag'].present?
conditions = QueryState.records(params, 'conditions')
drills = QueryState.records(params, 'drill')
raise ArgumentError unless drills.length <= 100
drills.each do |entry|
raise ArgumentError unless (entry.keys - ['field', 'grain', 'value']).empty?
condition = entry.merge('operator' => 'axis_eq')
conditions << condition unless conditions.include?(condition)
end
raise ArgumentError unless conditions.is_a?(Array) && conditions.length <= 100
values = QueryState.records(params, 'condition_values')
@@ -114,7 +179,13 @@ class CountsPopulation
raise ArgumentError unless index.between?(0, conditions.length - 1)
raise ArgumentError unless conditions[index]['values'] == true
end
evaluated = []
conditions.each_with_index do |entry, index|
if entry['operator'] == 'axis_eq'
next if evaluated.include?(entry)
evaluated << entry
end
if entry.key?('values')
raise ArgumentError unless entry['values'] == true
@@ -123,30 +194,7 @@ class CountsPopulation
end
apply_condition(entry)
end
drills = QueryState.records(params, 'drill')
raise ArgumentError unless drills.is_a?(Array) && drills.length <= 100
drills.each do |entry|
key = entry.fetch('field')
if key == 'tags'
@scope = @scope.where(snapshot_membership(entry.fetch('value')))
elsif key == 'changed_field'
raise ArgumentError unless history && definition.changes.include?(entry['value'])
@scope = @scope.where(Arel.sql(changed_sql(entry['value'])))
elsif definition.fields[key] == :datetime && entry['grain']
@scope = @scope.where(definition.expression(key) => nil) if entry['value'].nil?
if entry['value']
expression = CountsAxes.time_expression(definition.expression(key),
entry['grain'], timezone)
@scope = @scope.where("#{ expression } = ?", entry['value'])
end
else
raise ArgumentError unless definition.axes(history).include?(key)
@scope = @scope.where(definition.expression(key) => entry['value'])
end
end
end
def apply_condition(entry)
@@ -168,27 +216,28 @@ class CountsPopulation
def apply_tags
raise ArgumentError unless definition.resource == 'posts'
@scope = condition_scope(@scope, 'tags')
if params['tag'].present?
@scope = @scope.where(snapshot_membership(
"#{ params.fetch('kind', 'internal') }:#{ params['tag'] }"))
end
return unless params['tags'].present?
refs = PostTagSearch.parse_references(params, params['tags'].split)
raw = params['tags']
literals = PostTagSearch.literals(nil, raw, references: refs)
tokens = raw.split.reject { _1.downcase == 'type:or' }
locale = Locale.find_by(code: params.fetch('locale', 'ja'))
raise ArgumentError unless locale
literals = PostTagSearch.literals(locale, tokens.join(' '))
identities = PostTagSearch.references_for(locale, literals)
predicates = literals.map do |literal|
reference = literal[:reference]
expression = if reference
parts = { 'internal' => reference['tag_id'],
'external' => reference['external_tag_id'] }.filter_map do |kind, id|
snapshot_membership("#{ kind }:#{ id }") if id
end
parts.empty? ? 'FALSE' : "(#{ parts.join(' OR ') })"
else
@condition ||= CountsCondition.new(self)
@condition.tag_name_match(literal[:name])
end
# Resolve current names/aliases exactly as normal post search does. Only
# membership comes from history; snapshot names never identify a tag.
internal, external = identities.fetch(literal[:name])
parts = { 'internal' => internal, 'external' => external }.flat_map do |kind, ids|
Array(ids).map { |id| snapshot_membership("#{ kind }:#{ id }") }
end
expression = parts.empty? ? 'FALSE' : "(#{ parts.join(' OR ') })"
literal[:negative] ? "NOT #{ expression }" : expression
end
return if predicates.empty?
+18 -3
ファイルの表示
@@ -105,12 +105,25 @@ class CountsResourceDefinition
def time_column = 'created_at'
def column(key)
raise ArgumentError unless fields.key?(key)
# Keep the V1 Counts field name at the API boundary only.
return 'linked_tag_ids' if resource == 'nico_tags' && key == 'linked_tags'
{ 'resource_id' => keys.first, 'user' => 'created_by_user_id' }.fetch(key, key)
end
def expression(key, row = 'v') = "#{ row }.#{ column(key) }"
def changes = fields.keys - COMMON.keys - ['updated_at', 'updated_by_user_id']
def expression(key, row = 'v')
if resource == 'nico_tags' && key == 'name'
return "(SELECT CONCAT(external_tags.platform, ':', external_tags.name) " +
"FROM external_tags WHERE external_tags.id = #{ row }.tag_id)"
end
"#{ row }.#{ column(key) }"
end
def changes
columns = fields.keys - COMMON.keys - ['updated_at', 'updated_by_user_id']
resource == 'nico_tags' ? columns - ['name'] : columns
end
def axes(history)
fields.reject { |_key, type| type == :json }.keys +
(resource == 'posts' ? ['tags'] : []) + (history ? ['changed_field'] : [])
@@ -118,7 +131,9 @@ class CountsResourceDefinition
def metadata
fields.map do |key, type|
{ key:, type:, nullable: model.columns_hash.fetch(column(key)).null,
nullable = (resource == 'nico_tags' && key == 'name') ||
model.columns_hash.fetch(column(key)).null
{ key:, type:, nullable:,
operators: OPERATORS.fetch(type, []) }
end
end
+3 -1
ファイルの表示
@@ -201,6 +201,8 @@ class CountsStateQuery
end
def listing
external_name = CountsResourceDefinition.new('nico_tags')
.expression('name', 'external_version')
catalog = <<~SQL
, tag_counts AS (
SELECT membership.kind, membership.tag_id, COUNT(*) AS post_count
@@ -209,7 +211,7 @@ class CountsStateQuery
SELECT 'internal' AS kind, tag_id, name, names,
deprecated_at FROM tag_at WHERE event_type <> 'discard'
UNION ALL
SELECT tc.kind, tc.tag_id, external_version.name, JSON_OBJECT() AS names,
SELECT tc.kind, tc.tag_id, #{ external_name } AS name, JSON_OBJECT() AS names,
NULL AS deprecated_at
FROM tag_counts tc
LEFT JOIN tag_at internal_version
+3 -1
ファイルの表示
@@ -57,7 +57,7 @@ class CountsTableExport
def body(progress: nil, phase: nil)
Enumerator.new do |output|
ApplicationRecord.transaction(isolation: :repeatable_read) do
CountsExporter.with_read_snapshot do
processed = 0
started = false
first = true
@@ -253,6 +253,8 @@ class CountsTableExport
end
def summary_value(value, index, page)
return CountsAxes::MASKED_TEXT if value == CountsAxes::MASKED_VALUE
field = @query.axes.all.fetch(index).fetch('field')
return @labels.fetch('bot', '') if field == 'user' && value.nil?
if ['language_code', 'source_language_code'].include?(field)
+114 -42
ファイルの表示
@@ -1,10 +1,13 @@
# Common entrypoint for six version resources; legacy endpoints keep their contracts.
class CountsTableQuery
SQL_TIMEOUT_SECONDS = 30
SQL_HINTS = ["MAX_EXECUTION_TIME(#{ SQL_TIMEOUT_SECONDS * 1000 })",
"SET_VAR(optimizer_switch='semijoin=off')"].freeze
MAX_COLUMNS = 100
class TooManyColumns < ArgumentError; end
KEYS = ['resource', 'definition', 'mode', 'at', 'execution_at', 'from', 'through',
'timezone', 'rows', 'cols', 'conditions', 'drill', 'tags', 'tag_refs', 'match',
'timezone', 'rows', 'cols', 'conditions', 'drill', 'tags', 'match',
'tag', 'kind', 'resource_id', 'user_id', 'page', 'limit', 'sort', 'locale',
'measures', 'column_order', 'condition_values', 'name_scope'].freeze
attr_reader :population, :axes, :page, :limit, :mode
@@ -16,7 +19,9 @@ class CountsTableQuery
params = CountsDefaults.for(params).slice(*KEYS).merge(params)
@params = params
@population = CountsPopulation.new(params)
@mode = params.fetch('mode', 'summary')
validate_summary_conditions!
@population = CountsPopulation.new(params, authorisation_flg: mode != 'summary')
@definition = population.definition
@connection = @definition.model.connection
@axes = CountsAxes.new(population)
@@ -47,23 +52,30 @@ class CountsTableQuery
result = metadata.merge(mode == 'summary' ? summary : details)
visible = mode == 'summary' ? result[:visible_output_count] : result[:count]
totals = output_totals(visible)
result.merge(visible_output_count: visible,
response = result.merge(visible_output_count: visible,
**totals,
output_count_scope: 'before_authorisation',
output_count_unit: 'regular_result_rows',
page_grand_total_repeated: mode == 'summary' && axes.rows.any? &&
result[:group_count] > limit)
if mode == 'summary'
response.except!(:excluded_count, :excluded_count_scope)
response[:column_values_editable] = HistoryAccess.admin? ||
!result.fetch(:column_values, []).flatten.include?(CountsAxes::MASKED_VALUE)
end
response
end
end
def metadata
{ population_count: nil, population_count_computed: false,
{ condition_labels: condition_labels, population_count: nil, population_count_computed: false,
excluded_count: nil, excluded_count_scope: 'current_query',
population_count_scope: 'all_recorded_versions',
page:, limit:, execution_at: population.at.iso8601(6), at: @params['at'].presence,
definition: population.history ? 'history' : 'latest',
timezone: population.timezone.identifier, rows: axes.rows, cols: axes.columns,
fields: @definition.metadata,
masked_value: HistoryAccess.admin? ? nil : CountsAxes::MASKED_VALUE,
count: record_counts[:visible] }
end
@@ -75,19 +87,9 @@ class CountsTableQuery
cells: page == 1 ? [{ values: [], count: record_counts[:visible] }] : [],
axis_labels: {} }
end
expressions = axes.expressions
selected = expressions.each_with_index.map do |expression, index|
"#{ expression } AS a#{ index }"
end
source = population.scope
source = summary_population.scope
source = source.joins(axes.joins) unless axes.joins.empty?
sql = if expressions.empty?
source.except(:select).select('COUNT(DISTINCT v.id) AS count').to_sql
else
source.except(:select).group(*expressions.map { Arel.sql(_1) })
.select(Arel.sql((selected + ['COUNT(DISTINCT v.id) AS count']).join(', ')))
.to_sql
end
sql = axes.grouped_sql(source)
prefix = "WITH cells AS (#{ sql })"
column_values = limited_columns(prefix)
row_keys = axes.rows.each_index.map { "a#{ _1 }" }
@@ -128,7 +130,7 @@ class CountsTableQuery
numbered AS (SELECT row_groups.*,
ROW_NUMBER() OVER (ORDER BY #{ order_rows.join(', ') }) AS position FROM row_groups),
group_totals AS (SELECT COUNT(*) AS group_count FROM row_groups)
SELECT /*+ MAX_EXECUTION_TIME(10000) */ cells.*,
SELECT /*+ #{ SQL_HINTS.join(' ') } */ cells.*,
page_rows.position AS counts_position, group_totals.group_count AS counts_group_count
FROM group_totals LEFT JOIN numbered page_rows
ON page_rows.position > #{ (page - 1) * limit }
@@ -165,7 +167,7 @@ class CountsTableQuery
other_count: axes.rows.length, direction: :column)
.map { _1.merge(kind: 'grand_total') }
end
totals = CountsPivotTotals.new(population, axes, row_headers, column_headers).call
totals = CountsPivotTotals.new(summary_population, axes, row_headers, column_headers).call
{ group_count: total,
visible_output_count: axes.rows.empty? ? 1 : total,
column_values:, row_headers:, column_headers:, pivot_totals: totals,
@@ -175,11 +177,12 @@ class CountsTableQuery
end
def details
query = population.scope.select(Arel.sql(
"(#{ population.latest_sql }) AS counts_latest"))
query = population.scope.optimizer_hints(*SQL_HINTS).select(Arel.sql(
"(#{ population.current_latest_sql }) AS counts_latest"))
order = sort_sql { detail_expression(_1) }
query = query.order(Arel.sql((order + ['v.id ASC']).join(', ')))
.limit(limit).offset((page - 1) * limit)
query = query.preload(:external_tag) if @definition.resource == 'nico_tags'
rows = query.to_a
if ['posts', 'tags'].include?(@definition.resource)
point = population.history ? population.at :
@@ -218,21 +221,39 @@ class CountsTableQuery
private
# Private predicates require visibility independently of their operator.
# Ordering/measure state must not select a server-only masked coordinate.
def validate_summary_conditions!
return if mode != 'summary' || HistoryAccess.admin?
['column_order', 'measures'].each do |key|
# Coordinates are compared only to already masked expressions, never raw
# values. Public coordinates remain sortable; a guessed private ID cannot match.
raise CountsAxes::PrivateCondition if QueryState.records(@params, key).any? do |record|
record['value'] == CountsAxes::MASKED_VALUE
end
end
end
def unrestricted_population
@unrestricted_population ||= CountsPopulation.new(
@params.merge('execution_at' => population.at.iso8601(6)), authorisation_flg: false)
end
def summary_population
population
end
def record_counts
return @record_counts if @record_counts
if HistoryAccess.admin?
if HistoryAccess.admin? || mode == 'summary'
total = population.scope.except(:select)
.optimizer_hints('MAX_EXECUTION_TIME(10000)').count
.optimizer_hints(*SQL_HINTS).count
return @record_counts = { total:, visible: total, denied: 0 }
end
total, denied = unrestricted_population.scope.except(:select)
.optimizer_hints('MAX_EXECUTION_TIME(10000)').pick(Arel.sql('COUNT(*)'),
.optimizer_hints(*SQL_HINTS).pick(Arel.sql('COUNT(*)'),
Arel.sql("COALESCE(SUM(CASE WHEN #{ HistoryAccess.sql(@definition.model, 'v') } " +
'THEN 0 ELSE 1 END), 0)'))
# Related/previous-version predicates may differ under public authority.
@@ -242,14 +263,17 @@ class CountsTableQuery
total.to_i - denied.to_i
else
population.scope.except(:select)
.optimizer_hints('MAX_EXECUTION_TIME(10000)').count
.optimizer_hints(*SQL_HINTS).count
end
@record_counts = { total: total.to_i, denied: denied.to_i, visible: }
end
def output_totals(visible_rows)
def output_totals visible_rows
return @output_totals if @output_totals
if mode == 'summary'
return @output_totals = { output_count: visible_rows, excluded_count: nil }
end
if HistoryAccess.admin?
return @output_totals = { output_count: visible_rows, excluded_count: 0 }
end
@@ -281,7 +305,7 @@ class CountsTableQuery
prefix = "WITH cells AS (#{ sql })"
@connection.select_value(<<~SQL).to_i
#{ prefix }
SELECT /*+ MAX_EXECUTION_TIME(10000) */ COUNT(*) FROM (
SELECT /*+ #{ SQL_HINTS.join(' ') } */ COUNT(*) FROM (
SELECT #{ keys.join(', ') } FROM cells GROUP BY #{ keys.join(', ') }
) output_groups
SQL
@@ -289,7 +313,7 @@ class CountsTableQuery
def locale = @locale ||= Locale.find_by!(code: @params.fetch('locale', 'ja'))
def limited_columns(prefix)
def limited_columns prefix
keys = axes.columns.each_index.map { "a#{ axes.rows.length + _1 }" }
return [[]] if keys.empty?
@@ -307,7 +331,7 @@ class CountsTableQuery
end
values = @connection.select_rows(<<~SQL)
#{ prefix }
SELECT /*+ MAX_EXECUTION_TIME(10000) */ DISTINCT #{ keys.join(', ') } FROM cells
SELECT /*+ #{ SQL_HINTS.join(' ') } */ DISTINCT #{ keys.join(', ') } FROM cells
ORDER BY #{ order } LIMIT #{ MAX_COLUMNS + 1 }
SQL
raise TooManyColumns if values.length > MAX_COLUMNS
@@ -315,9 +339,13 @@ class CountsTableQuery
values
end
def measure_relation(values)
row_expressions = axes.expressions.first(axes.rows.length)
source = population.scope.except(:select, :order)
def measure_relation values
source = summary_population.scope.except(:select, :order)
# Omitted multi-value column axes still determine full-cell membership.
membership = summary_population.scope.except(:select, :order)
all_joins = axes.joins
membership = membership.joins(all_joins) unless all_joins.empty?
source = source.where("v.id IN (#{ membership.select('v.id').distinct.to_sql })")
fields = axes.rows + axes.columns.first(values.length)
joins = axes.joins(fields.map { _1.fetch('field') })
source = source.joins(joins) unless joins.empty?
@@ -325,17 +353,23 @@ class CountsTableQuery
expression = axes.expressions[axes.rows.length + index]
source = source.where("(#{ expression }) <=> ?", value)
end
selected = row_expressions.each_with_index.map { |expression, index|
"#{ expression } AS a#{ index }"
} + ['COUNT(DISTINCT v.id) AS count']
source = source.group(*row_expressions.map { Arel.sql(_1) }) if row_expressions.any?
source.select(Arel.sql(selected.join(', '))).to_sql
axes.grouped_sql(source, axes.rows.each_index.to_a)
end
def axis_order_sql(key, direction = 'asc', qualifier: nil)
def axis_order_sql key, direction = 'asc', qualifier: nil
column = qualifier ? "#{ qualifier }.#{ key }" : key
expression = column
if axes.all.fetch(key.delete_prefix('a').to_i).fetch('field') == 'category'
field = axes.all.fetch(key.delete_prefix('a').to_i).fetch('field')
if !HistoryAccess.admin? && [:integer, :boolean].include?(@definition.fields[field])
masked = @connection.quote(CountsAxes::MASKED_VALUE)
expression = "CASE WHEN #{ column } <=> #{ masked } THEN NULL " +
"ELSE CAST(#{ column } AS SIGNED) END"
return "#{ column } IS NULL ASC, (#{ column } <=> #{ masked }) ASC, " +
"#{ expression } #{ direction.upcase }"
end
if field == 'event_type'
expression = event_order_sql(column)
elsif field == 'category'
# Tag's enum preserves the same category order as TagsController.
cases = Tag.categories.keys.each_with_index.map { |category, index|
"WHEN #{ @connection.quote(category) } THEN #{ index }"
@@ -345,7 +379,7 @@ class CountsTableQuery
"#{ column } IS NULL ASC, #{ expression } #{ direction.upcase }"
end
def measure_values(key)
def measure_values key
records = QueryState.records(@params, 'measures').select { _1['key'] == key }
raise ArgumentError if records.empty?
raise ArgumentError unless records.map { _1['axis'] }.sort == (0...records.length).to_a
@@ -359,7 +393,8 @@ class CountsTableQuery
end
def detail_expression(key)
return "(#{ population.latest_sql })" if key == 'latest'
return "(#{ population.current_latest_sql })" if key == 'latest'
return event_order_sql(@definition.expression(key)) if key == 'event_type'
return @definition.name_expression(key) if @definition.resource == 'tags' &&
@definition.name_column(key)
aliases = { 'tags' => 'tags_json', 'parents' => @definition.resource == 'posts' ?
@@ -371,6 +406,11 @@ class CountsTableQuery
"CAST(#{ expression } AS CHAR) COLLATE utf8mb4_bin"
end
def event_order_sql(expression)
"CASE #{ expression } WHEN 'create' THEN 0 WHEN 'update' THEN 1 " +
"WHEN 'discard' THEN 2 WHEN 'restore' THEN 3 ELSE 4 END"
end
def sort_sql
@sort.flat_map do |key, direction|
expression = yield key
@@ -378,6 +418,25 @@ class CountsTableQuery
end
end
def condition_labels
entries = QueryState.records(@params, 'conditions').select { _1['operator'] == 'axis_eq' }
entries += QueryState.records(@params, 'drill')
entries.group_by { _1['field'] }.to_h do |field, conditions|
values = conditions.map { _1['value'] }.compact.uniq
labels = case field
when 'tags' then tag_labels(values)
when 'parent_tag_ids' then parent_tag_labels(values)
when 'user'
actors = HistoryAccess.visible(population.scope, 'v').except(:select, :order)
.select('v.created_by_user_id')
User.where(id: values).where(id: actors)
.pluck(:id, :name).to_h.transform_keys(&:to_s)
else {}
end
[field, labels]
end
end
def axis_labels(records, column_values)
result = {}
@axis_names = {}
@@ -386,7 +445,7 @@ class CountsTableQuery
if index >= axes.rows.length
values += column_values.map { _1[index - axes.rows.length] }
end
values = values.compact.uniq
values = values.compact.uniq - [CountsAxes::MASKED_VALUE]
labels = case axis['field']
when 'user'
User.where(id: values).pluck(:id, :name).to_h.transform_keys(&:to_s)
@@ -396,6 +455,7 @@ class CountsTableQuery
labels = tag_labels(values)
@axis_names["a#{ index }"] = @tag_names
labels
when 'parent_tag_ids' then parent_tag_labels(values)
else {}
end
result["a#{ index }"] = labels
@@ -403,6 +463,17 @@ class CountsTableQuery
result
end
def parent_tag_labels(values)
ids = values.flat_map { _1.to_s.split }.uniq
tag_labels(ids.map { "internal:#{ _1 }" })
values.to_h do |value|
names = value.to_s.split.map do |id|
@tag_names.fetch("internal:#{ id }", {}).dig(locale.language_code, 'name')
end
[value.to_s, names.compact.join(' ')]
end
end
def tag_labels(values)
internal = values.grep(/\Ainternal:/).map { _1.split(':').last.to_i }
external = values.grep(/\Aexternal:/).map { _1.split(':').last.to_i }
@@ -416,10 +487,11 @@ class CountsTableQuery
table = model.table_name
scope = HistoryAccess.visible(model.where(tag_id: ids)).where('created_at <= ?', point)
scope = scope.includes(:external_tag) if model == NicoTagVersion
bound = @connection.quote(point)
if population.history && @definition.resource == 'posts'
key = kind == 'internal' ? 'tag_id' : 'external_tag_id'
pairs = population.scope.except(:select, :order).joins(
pairs = HistoryAccess.visible(population.scope, 'v').except(:select, :order).joins(
"JOIN JSON_TABLE(v.tags_json, '$[*]' COLUMNS (tag_id BIGINT PATH '$.#{ key }', " +
"version_no INT PATH '$.version_no' NULL ON EMPTY)) recorded_tag")
.where('recorded_tag.tag_id IN (?)', ids).group('recorded_tag.tag_id')
+2 -2
ファイルの表示
@@ -1,4 +1,5 @@
class NicoTagVersionRecorder < VersionRecorder
EVENT_TYPES = ['create', 'update'].freeze
def self.record! external_tag:, event_type:, created_by_user:
new(external_tag:, event_type:, created_by_user:).record!
end
@@ -14,8 +15,7 @@ class NicoTagVersionRecorder < VersionRecorder
def record_key = :external_tag
def snapshot_attributes
{ name: "#{ @record.platform }:#{ @record.name }",
linked_tags: @record.snapshot_linked_tag_names.join(' ') }
{ linked_tag_ids: @record.snapshot_linked_tag_ids.join(' ') }
end
def tracks_version_no_on_record? = false
+21 -33
ファイルの表示
@@ -1,42 +1,30 @@
# Shared literal parsing and reference resolution for current and snapshot post search.
class PostTagSearch
def self.parse_references(raw, literals)
return nil unless raw.key?('tag_refs')
references = QueryState.records(raw, 'tag_refs')
valid = references.is_a?(Array) && references.all? do |reference|
next false unless reference.is_a?(Hash) && reference['name'].is_a?(String)
ids = [reference['tag_id'], reference['external_tag_id']].compact
((ids.any? || reference['missing'] == true) &&
ids.all? { _1.is_a?(Integer) && _1.positive? } &&
(!reference.key?('missing') || (reference['missing'] == true && ids.empty?)) &&
(!reference.key?('negative') || [true, false].include?(reference['negative'])))
end
raise ArgumentError unless valid
references.map { _1['name'] } == literals ? references : nil
end
def self.literals(locale, raw, references: nil)
raw.split.each_with_index.filter_map do |token, index|
next if token.downcase == 'type:or'
reference = references&.[](index)
name = token.sub(/\Anot:/i, '')
name = TagName.canonicalise(locale, name).first unless reference || locale.nil?
negative = reference&.key?('negative') ? reference['negative'] :
token.downcase.start_with?('not:')
{ name:, negative:, reference: }
def self.literals(locale, raw)
tokens = raw.split.reject { _1.downcase == 'type:or' }
names = tokens.map { _1.sub(/\Anot:/i, '') }
registered = TagName.where(language_code: locale.language_code, name: names)
.pluck(:name, :tag_id).to_h
primary = TagName.where(language_code: locale.language_code,
tag_id: registered.values, primary_flg: true).pluck(:tag_id, :name).to_h
tokens.zip(names).map do |token, name|
{ name: primary[registered[name]] || name,
negative: token.downcase.start_with?('not:') }
end
end
def self.references(locale, literal)
reference = literal[:reference]
return [reference['tag_id'], reference['external_tag_id']] if reference
references_for(locale, [literal]).fetch(literal[:name])
end
[TagName.where(language_code: locale.language_code, name: literal[:name])
.distinct.pluck(:tag_id),
ExternalTag.where("CONCAT(platform, ':', name) = ?", literal[:name]).pluck(:id)]
def self.references_for(locale, literals)
names = literals.map { _1[:name] }.uniq
internal = TagName.where(language_code: locale.language_code, name: names)
.pluck(:name, :tag_id).group_by(&:first)
external = names.empty? ? {} : ExternalTag.where(
"CONCAT(platform, ':', name) IN (?)", names)
.pluck(Arel.sql("CONCAT(platform, ':', name)"), :id).group_by(&:first)
names.to_h { |name| [name, [internal.fetch(name, []).map(&:last).uniq,
external.fetch(name, []).map(&:last).uniq]] }
end
end
+5 -5
ファイルの表示
@@ -1,10 +1,9 @@
# Public query grammar shared with frontend/src/lib/queryState.ts.
class QueryState
SCHEMAS = {
'conditions' => ['field', 'operator', 'value', 'end', 'blank', 'input', 'values'],
'conditions' => ['field', 'operator', 'value', 'end', 'blank', 'input', 'values', 'grain', 'value_type'],
'condition_values' => ['condition', 'value', 'value_type'],
'drill' => ['field', 'grain', 'value', 'value_type'],
'tag_refs' => ['name', 'tag_id', 'external_tag_id', 'negative', 'missing'],
'column_order' => ['value', 'value_type'],
'measures' => ['key', 'axis', 'value', 'value_type'] }.freeze
@@ -41,11 +40,11 @@ class QueryState
_root, index, field = name.split('.')
record = records[index.to_i] ||= {}
if ['blank', 'negative', 'missing', 'values'].include?(field)
if ['blank', 'values'].include?(field)
raise ArgumentError unless ['0', '1'].include?(value)
value = value == '1'
elsif ['tag_id', 'external_tag_id', 'axis', 'condition'].include?(field)
elsif ['axis', 'condition'].include?(field)
raise ArgumentError unless value.match?(/\A[0-9]{1,18}\z/)
value = value.to_i
@@ -57,7 +56,8 @@ class QueryState
records.sort.map do |_index, record|
type = record.delete('value_type')
if ['drill', 'measures', 'column_order', 'condition_values'].include?(key)
if (['drill', 'measures', 'column_order', 'condition_values'].include?(key) ||
(key == 'conditions' && record['operator'] == 'axis_eq'))
raise ArgumentError unless ['null', 'string', 'number', 'boolean'].include?(type)
raise ArgumentError unless type == 'null' || record.key?('value')
+255
ファイルの表示
@@ -0,0 +1,255 @@
require 'cgi'
require 'digest'
require 'erb'
require 'pathname'
require 'uri'
# A deliberately small current-state projection, always under public authority.
class StaticMirror
ORIGIN = 'https://hub.nizika.monster'.freeze
PUBLICATION_POLICY = 'static-mirror-public-projection-1'.freeze
DATA_AT_BASIS = 'capture_started_at'.freeze
RESOURCE_LIMIT = 100
NAME_LIMIT = 2_000
MAX_BYTES = 8.megabytes
EXCLUSIONS = [
'historical_state', 'unversioned_resources', 'tag_descriptions', 'help', 'materials',
'external_tags', 'parent_relations', 'tag_sections', 'users', 'audit_data',
'authentication', 'editing', 'images', 'videos', 'resources_beyond_limits'].freeze
class TooLarge < StandardError; end
class UnsafeUrl < StandardError; end
attr_reader :data_at, :authority, :counts
def initialize(locale:, origin: ORIGIN)
@locale = locale
@origin = validated_url(origin).sub(%r{/+\z}, '')
end
def export(&progress)
@files = { }
@bytes = 0
HistoryAccess.set(user: nil) do
capture
progress&.call(0, @posts.length + @tags.length)
generate_pages(&progress)
raise HistoryAccess::Forbidden unless self.class.authorised_artifact?(authority)
generated_at = Time.current.utc
manifest = { format_version: 1, generated_at: generated_at.iso8601(6),
data_at:, data_at_basis: DATA_AT_BASIS,
locale: @locale.code, resources: counts, exclusions: EXCLUSIONS,
fields: { posts: ['id', 'title', 'url', 'tags'], tags: ['id', 'category', 'names'] },
limits: { posts: RESOURCE_LIMIT, tags: RESOURCE_LIMIT, names: NAME_LIMIT,
bytes: MAX_BYTES },
external_assets: 'not_included', checksum_scope: 'all_files_except_manifest.json',
files: @files.map { |path, data| { path:, sha256: Digest::SHA256.hexdigest(data) } } }
add_file('manifest.json', JSON.pretty_generate(manifest))
entries = @files.map do |path, data|
MaterialZipExporter::Entry.new(path: "archive/#{ path }", data:, mtime: generated_at)
end
bytes = MaterialZipExporter::ZipWriter.write(entries)
raise TooLarge if bytes.bytesize > MAX_BYTES
bytes
end
end
# Static distribution has no admin mode or Counts aggregate exception.
# Recheck the exact public projection, including edits without a new discard.
def self.authorised_artifact?(metadata)
return false unless metadata['publication_policy'] == PUBLICATION_POLICY
ids = metadata.values_at('post_ids', 'tag_ids')
unless ids.all? { |values|
values.is_a?(Array) && values.length <= RESOURCE_LIMIT &&
values.all? { _1.is_a?(Integer) && _1.positive? }
}
return false
end
HistoryAccess.set(user: nil) do
ApplicationRecord.uncached do
CountsExporter.with_read_snapshot do
Locale.where(code: metadata['locale'], deprecated_at: nil).exists? &&
metadata['history_revision'] == HistoryAccess.revision &&
metadata['content_digest'] == projection_digest(
read_projection(post_ids: ids[0], tag_ids: ids[1]))
end
end
end
rescue TooLarge
false
end
def self.read_projection(post_ids: nil, tag_ids: nil)
posts = confirmed(Post, PostVersion, :post_versions)
tags = confirmed(Tag, TagVersion, :tag_versions).where(deprecated_at: nil)
posts = posts.where(id: post_ids) unless post_ids.nil?
tags = tags.where(id: tag_ids) unless tag_ids.nil?
posts = posts.order(:id).limit(RESOURCE_LIMIT).select(:id, :version_no, :title, :url).to_a
tags = tags.order(:id).limit(RESOURCE_LIMIT).select(:id, :version_no, :category).to_a
names = TagName.where(tag_id: tags.map(&:id)).order(:tag_id, :language_code, :id)
.limit(NAME_LIMIT + 1).pluck(:tag_id, :language_code, :script_code, :name, :primary_flg)
raise TooLarge if names.length > NAME_LIMIT
relations = PostTag.where(post_id: posts.map(&:id), tag_id: tags.map(&:id))
.order(:post_id, :tag_id).pluck(:post_id, :tag_id)
[posts, tags, names, relations]
end
def self.confirmed(model, version_model, association)
# Requiring a matching immutable revision closes the legacy unversioned gap.
HistoryAccess.current(model.all, version_model).joins(association)
.where("#{ version_model.table_name }.version_no = #{ model.table_name }.version_no")
end
def self.projection_digest(projection)
posts, tags, names, relations = projection
Digest::SHA256.hexdigest(JSON.generate(
[posts.map(&:attributes), tags.map(&:attributes), names, relations]))
end
private_class_method :confirmed
private
def capture
# Server clock at capture entry; MySQL establishes the read view later.
@data_at = Time.current.utc.iso8601(6)
CountsExporter.with_read_snapshot do
projection = self.class.read_projection
@posts, @tags, names, @relations = projection
@authority = { 'publication_policy' => PUBLICATION_POLICY,
'locale' => @locale.code,
'history_revision' => HistoryAccess.revision,
'post_ids' => @posts.map(&:id), 'tag_ids' => @tags.map(&:id),
'content_digest' => self.class.projection_digest(projection) }
@names = names.group_by(&:first)
@counts = { posts: @posts.length, tags: @tags.length }
end
end
def generate_pages
assets = Rails.root.join('app/views/static_mirror')
['archive.css', 'archive.js'].each do |name|
add_file("assets/#{ name }", assets.join(name).read)
end
add_file('robots.txt', "User-agent: *\nDisallow: /\n")
add_file('archive-config.js', 'window.archiveConfig = { origin: null };')
page('index.html', 'Static Mirror 生成機能', '', search_form + limitations)
page("#{ @locale.code }/posts/index.html", 'Post', '/posts', post_list(@posts))
page("#{ @locale.code }/tags/index.html", 'Tag', '/tags', tag_list(@tags))
processed = 0
@posts.each do |post|
tags = @tags.select { |tag| @relations.include?([post.id, tag.id]) }
content = "<p>#{ external_link(post.url) }</p>#{ tag_list(tags) }"
page(resource_path('posts', post.id), post.title.presence || post.id.to_s,
"/posts/#{ post.id }", content)
processed += 1
yield(processed, @posts.length + @tags.length) if block_given?
end
@tags.each do |tag|
posts = @posts.select { |post| @relations.include?([post.id, tag.id]) }
names = @names.fetch(tag.id, []).map do |_, language, script, name, _primary|
"<li><span lang=\"#{ h(language) }\">#{ h(name) }</span> (#{ h(script) })</li>"
end.join
content = "<p>#{ h(tag.category) }</p><ul>#{ names }</ul>#{ post_list(posts) }"
page(resource_path('tags', tag.id), tag_name(tag), "/tags/#{ tag.id }", content)
processed += 1
yield(processed, @posts.length + @tags.length) if block_given?
end
index = @posts.map { |post|
tag_ids = @relations.filter_map { |post_id, tag_id| tag_id if post_id == post.id }
names = tag_ids.flat_map { |id| @names.fetch(id, []).map { _1[3] } }.uniq
{ path: resource_path('posts', post.id), title: post.title.presence || post.id.to_s,
names: }
} + @tags.map { |tag|
{ path: resource_path('tags', tag.id), title: tag_name(tag),
names: @names.fetch(tag.id, []).map { _1[3] } }
}
add_file('search-index.json', JSON.generate(index))
end
def page(path, title, original_path, content)
@path = path
@title = title
@original_path = "/#{ @locale.code }#{ original_path }"
@content = content
template = Rails.root.join('app/views/static_mirror/page.html.erb').read
add_file(path, ERB.new(template).result(binding))
end
def add_file(path, data)
@bytes += data.bytesize
raise TooLarge if @bytes > MAX_BYTES
@files[path] = data
end
def resource_path(resource, id) = "#{ @locale.code }/#{ resource }/#{ id }/index.html"
def relative(path)
Pathname.new(path).relative_path_from(Pathname.new(@path).dirname).to_s
end
def h(value) = CGI.escapeHTML(value.to_s)
def tag_name(tag)
@names.fetch(tag.id, []).find { _1[1] == @locale.language_code && _1[4] }&.fetch(3) ||
tag.id.to_s
end
# Links are resolved after the page path is known; only generated numeric IDs enter paths.
def post_list(posts)
'<h2>Post</h2><ul>' + posts.map { |post|
title = post.title.presence || post.id.to_s
"<li><a data-path=\"#{ resource_path('posts', post.id) }\">#{ h(title) }</a></li>"
}.join + '</ul>'
end
def tag_list(tags)
'<h2>Tag</h2><ul>' + tags.map { |tag|
"<li><a data-path=\"#{ resource_path('tags', tag.id) }\">#{ h(tag_name(tag)) }</a></li>"
}.join + '</ul>'
end
def linked_content
@content.gsub(/data-path="([^"]+)"/) { "href=\"#{ h(relative(Regexp.last_match(1))) }\"" }
end
def validated_url(value)
uri = URI.parse(value.to_s)
unless uri.is_a?(URI::HTTP) && uri.host.present? && uri.userinfo.nil?
raise UnsafeUrl
end
uri.to_s
rescue URI::InvalidURIError
raise UnsafeUrl
end
def external_link(value)
url = validated_url(value)
"<a href=\"#{ h(url) }\" target=\"_blank\" rel=\"noopener noreferrer\">#{ h(url) }</a>"
rescue UnsafeUrl
''
end
def search_form
'<form id="archive-search"><label>検索 <input name="q" type="search"></label>' +
'<button type="submit">検索</button></form><ul id="search-results"></ul>'
end
def limitations
<<~HTML
<h2>未対応機能</h2>
<p>Tag Description, Help, Material, 外部タグ, 親子関係, 区間, 履歴, ログイン, 作成, 編輯</p>
<h2>出力件数の制限</h2><p>Post: #{ RESOURCE_LIMIT }, Tag: #{ RESOURCE_LIMIT }</p>
<p>TagName: #{ NAME_LIMIT }, #{ MAX_BYTES } B</p>
<h2>外部画像や動画への依存</h2><p>ローカルへ取り込めていない資産: 外部画像や動画</p>
<h2>データ保護上の制約</h2>
<p>公開抑止、除票、非公開情報、認証情報、監査専用情報を ZIP に含めない。</p>
<p>今回は現在状態のみの対応</p>
HTML
end
end
+7
ファイルの表示
@@ -0,0 +1,7 @@
:root { color-scheme: light dark; }
body { max-width: 65rem; margin: auto; padding: 1rem; font: 1rem/1.6 sans-serif; }
header { border-bottom: 1px solid; padding-bottom: 1rem; }
nav { display: flex; flex-wrap: wrap; gap: 1rem; }
a, p, li { overflow-wrap: anywhere; }
li { margin-block: .4rem; }
input, button { font: inherit; }
+61
ファイルの表示
@@ -0,0 +1,61 @@
(() => {
const root = new URL ('../', document.currentScript.src)
const origin = window.archiveConfig?.origin
if (typeof origin === 'string')
{
try
{
const url = new URL (origin)
if (['http:', 'https:'].includes (url.protocol) && !url.username && !url.password)
{
url.search = ''
url.hash = ''
document.querySelectorAll ('[data-original]').forEach (link => {
link.href = url.href.replace (/\/+$/, '') + link.dataset.original
})
}
}
catch
{
// Each HTML retains its original link when configuration is absent or invalid.
}
}
const form = document.getElementById ('archive-search')
const results = document.getElementById ('search-results')
if (!form || !results) return
form.addEventListener ('submit', async event => {
event.preventDefault ()
results.replaceChildren ()
const query = (new FormData (form).get ('q') ?? '').toString ().toLocaleLowerCase ()
if (!query.trim ()) return
try
{
const response = await fetch (new URL ('search-index.json', root))
if (!response.ok) throw new Error ('Search index unavailable')
const entries = await response.json ()
for (const entry of entries)
{
const text = [entry.title, ...(entry.names ?? [])].join (' ').toLocaleLowerCase ()
if (!text.includes (query)) continue
const url = new URL (entry.path, root)
if (url.origin !== root.origin || !url.pathname.startsWith (root.pathname)) continue
const item = document.createElement ('li')
const link = document.createElement ('a')
link.setAttribute ('href', entry.path)
link.textContent = entry.title
item.append (link)
results.append (item)
}
}
catch
{
// Lists remain available if the index cannot be loaded.
for (const link of document.querySelectorAll ('nav a:not([data-original])'))
{
const item = document.createElement ('li')
item.append (link.cloneNode (true))
results.append (item)
}
}
})
})()
+28
ファイルの表示
@@ -0,0 +1,28 @@
<!doctype html>
<html lang="ja">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="robots" content="noindex, nofollow">
<meta name="referrer" content="no-referrer">
<meta http-equiv="Content-Security-Policy"
content="default-src 'none'; script-src 'self'; style-src 'self'; connect-src 'self'; form-action 'none'; base-uri 'none'">
<title><%= h(@title) %></title>
<link rel="stylesheet" href="<%= h(relative('assets/archive.css')) %>">
<script defer src="<%= h(relative('archive-config.js')) %>"></script>
<script defer src="<%= h(relative('assets/archive.js')) %>"></script>
</head>
<body>
<header>
<p>静的保存版 — データ基準時点: <time><%= h(data_at) %></time></p>
<nav>
<a href="<%= h(relative('index.html')) %>">Static Mirror 生成機能</a>
<a href="<%= h(relative("#{ @locale.code }/posts/index.html")) %>">Post</a>
<a href="<%= h(relative("#{ @locale.code }/tags/index.html")) %>">Tag</a>
<a href="<%= h(@origin + @original_path) %>" data-original="<%= h(@original_path) %>"
target="_blank" rel="noopener noreferrer">本家へのリンク</a>
</nav>
</header>
<main><h1><%= h(@title) %></h1><%= linked_content %></main>
</body>
</html>
+6
ファイルの表示
@@ -1,8 +1,14 @@
Rails.application.routes.draw do
get 'static_mirror', to: 'static_mirrors#show'
post 'static_mirror/jobs', to: 'static_mirrors#create'
get 'static_mirror/jobs/:id', to: 'static_mirrors#status'
get 'static_mirror/files/:id', to: 'static_mirrors#download'
get 'counts/definitions/:resource', to: 'counts_tables#definition'
get 'counts/definitions/:resource/candidates', to: 'counts_tables#candidates'
get 'counts/query', to: 'counts_tables#index'
post 'counts/export/jobs', to: 'counts_exports#create_job'
post 'counts/export/jobs/:token/download', to: 'counts_exports#start_download'
get 'counts/export/jobs/:token', to: 'counts_exports#job_status'
delete 'counts/export/jobs/:token', to: 'counts_exports#cancel_job'
post 'counts/export/prepare', to: 'counts_exports#prepare'
+5
ファイルの表示
@@ -0,0 +1,5 @@
class AddLinkedTagIdsToNicoTagVersions < ActiveRecord::Migration[8.0]
def change
add_column :nico_tag_versions, :linked_tag_ids, :text
end
end
+42
ファイルの表示
@@ -0,0 +1,42 @@
class MigrateLinkedTagsInNicoTagVersions < ActiveRecord::Migration[8.0]
class MigrationTagVersion < ActiveRecord::Base
self.table_name = 'tag_versions'
end
class MigrationNicoTagVersion < ActiveRecord::Base
self.table_name = 'nico_tag_versions'
end
def up
# 除票外部タグの恢復はないので除票履歴があれば全消しでよいはず.
execute <<~SQL
DELETE
t1
FROM
nico_tag_versions t1
INNER JOIN
nico_tag_versions t2
ON
t1.tag_id = t2.tag_id
AND t2.event_type = 'discard'
SQL
MigrationNicoTagVersion.find_each do |ntv|
linked_tag_ids = ntv.linked_tags.split.map { |name|
# 厳密には一意ぢゃないけどめんだぅだからこれでいいや.
MigrationTagVersion
.order(created_at: :desc, id: :desc)
.find_by("names ->> '$.ja.name' = ?", name)
&.tag_id
}.compact.uniq.sort.join(' ')
ntv.update_columns(linked_tag_ids:)
end
change_column_null :nico_tag_versions, :linked_tag_ids, false
end
def down
raise ActiveRecord::IrreversibleMigration, '戻せません.'
end
end
+6
ファイルの表示
@@ -0,0 +1,6 @@
class RemoveLinkedTagsFromNicoTagVersions < ActiveRecord::Migration[8.0]
def change
remove_column :nico_tag_versions, :linked_tags
remove_column :nico_tag_versions, :name
end
end
生成ファイル
+2 -3
ファイルの表示
@@ -10,7 +10,7 @@
#
# It's strongly recommended that you check this file into your version control system.
ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
ActiveRecord::Schema[8.0].define(version: 2026_11_08_020000) do
create_table "active_storage_attachments", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.string "name", null: false
t.string "record_type", null: false
@@ -301,10 +301,9 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
t.bigint "tag_id", null: false
t.integer "version_no", null: false
t.string "event_type", null: false
t.string "name", null: false
t.text "linked_tags", null: false
t.datetime "created_at", null: false
t.bigint "created_by_user_id"
t.text "linked_tag_ids", null: false
t.index ["created_at"], name: "index_nico_tag_versions_on_created_at"
t.index ["created_by_user_id", "created_at"], name: "index_nico_tag_versions_on_created_by_user_id_and_created_at", order: { created_at: :desc }
t.index ["tag_id", "created_at"], name: "index_nico_tag_versions_on_tag_id_and_created_at", order: { created_at: :desc }
+15 -2
ファイルの表示
@@ -9,8 +9,21 @@
# end
ApplicationRecord.transaction do
Language.find_or_create_by!(code: 'ain') { _1.name = 'ain' }
Script.find_or_create_by!(code: 'Kana') { _1.name = 'Kana' }
Language.find_or_create_by!(code: 'ja') { _1.name = '日本語' }
Script.find_or_create_by!(code: 'Jpan') { _1.name = '漢字、仮名文字およびラテン文字' }
Region.find_or_create_by!(code: 392) do
_1.alpha_code = 'JP'
_1.name = '日本国'
end
Locale.find_or_create_by!(code: 'ja') do
_1.language_code = 'ja'
_1.script_code = 'Jpan'
_1.region_code = 392
_1.name = '日本語'
end
Language.find_or_create_by!(code: 'ain') { _1.name = 'ヤウンモシルン アイヌ イタㇰ' }
Script.find_or_create_by!(code: 'Kana') { _1.name = 'カナモジ オヨビ ラテンモジ' }
end
post_url_sanitisation_rules = [
+43 -8
ファイルの表示
@@ -11,28 +11,63 @@ RSpec.describe ExternalTag, type: :model do
expect(external.reload.name).to eq('raw tag[]')
end
it 'deletes post associations without deleting posts or version history' do
it 'does not allow a persisted external tag to change its platform or name' do
external = create(:external_tag, name: 'immutable')
expect { external.update!(name: 'renamed') }
.to raise_error(ActiveRecord::RecordInvalid)
expect(external.errors[:name]).to include('は変更できません.')
external.reload
external.platform = nil
expect { external.save! }.to raise_error(ActiveRecord::RecordInvalid)
expect(external.errors[:platform]).to include('は変更できません.')
expect(external.reload).to have_attributes(platform: 'nico', name: 'immutable')
end
it 'rejects delete, which bypasses destroy callbacks' do
external = create(:external_tag)
expect { external.delete }.to raise_error(ActiveRecord::ReadOnlyRecord)
expect(external.reload).to be_persisted
end
it 'increments the counter through post associations without changing identity' do
external = create(:external_tag, name: 'counter')
post = create(:post)
expect {
PostExternalTag.create!(post:, external_tag: external)
}.to change { external.reload.post_count }.by(1)
expect(external.reload).to have_attributes(platform: 'nico', name: 'counter')
end
it 'does not destroy external tags or their post associations and history' do
external = create(:external_tag)
post = create(:post)
PostExternalTag.create!(post:, external_tag: external)
version = NicoTagVersionRecorder.record!(
external_tag: external, event_type: :create, created_by_user: nil)
external.destroy!
expect { external.destroy! }.to raise_error(ActiveRecord::RecordNotDestroyed)
expect(PostExternalTag.where(external_tag_id: external.id)).to be_empty
expect(post.reload.external_tags).to be_empty
expect(external.reload).to be_persisted
expect(PostExternalTag.exists?(post:, external_tag: external)).to be(true)
expect(post.reload.external_tags).to contain_exactly(external)
expect(version.reload.tag_id).to eq(external.id)
end
it 'deletes external links without deleting linked internal tags' do
it 'does not destroy an external tag or its links to internal tags' do
external = create(:external_tag)
tag = create(:tag)
NicoTagRelation.create!(nico_tag: external, tag:)
relation = NicoTagRelation.create!(nico_tag: external, tag:)
external.destroy!
expect { external.destroy! }.to raise_error(ActiveRecord::RecordNotDestroyed)
expect(NicoTagRelation.where(nico_tag_id: external.id)).to be_empty
expect(external.reload).to be_persisted
expect(relation.reload).to be_persisted
expect(Tag.exists?(tag.id)).to be(true)
end
end
+130
ファイルの表示
@@ -0,0 +1,130 @@
require 'rails_helper'
RSpec.describe HistoryAccess do
include_context 'Ainu locale'
let!(:tag) { create(:tag) }
it 'uses the complete tag-description identity and version order at equal timestamps' do
locale
rows = ['ja', 'ain'].flat_map do |language|
['create', 'discard', 'restore'].each_with_index.map do |event, index|
TagDescriptionVersion.create!(tag_id: tag.id, language_code: language,
version_no: index + 1, event_type: language == 'ain' ? 'update' : event,
body: "#{ language } #{ index }", created_at: Time.utc(2024))
end
end
described_class.set(user: nil) do
expect(described_class.visible(TagDescriptionVersion.where(id: rows.map(&:id))).pluck(:id))
.to match_array([rows[2], *rows.last(3)].map(&:id))
expect { described_class.require!(rows.first) }.to raise_error(HistoryAccess::Forbidden)
end
described_class.set(user: create(:user, :admin)) do
expect(described_class.visible(TagDescriptionVersion.where(id: rows.map(&:id))).count).to eq(6)
end
end
it 'invalidates public artifacts after a new discard and binds admin artifacts to their actor' do
public_metadata = described_class.set(user: nil) { described_class.authority }
admin = create(:user, :admin)
admin_metadata = described_class.set(user: admin) { described_class.authority }
TagVersion.create!(tag:, version_no: 1, event_type: 'discard', name: 'private',
category: 'general',
names: { ja: { name: 'private', script_code: 'Jpan',
auto_generated: false, aliases: [] } },
aliases: '', parent_tag_ids: '', created_at: Time.utc(2024))
described_class.set(user: nil) do
expect(described_class.authorised_artifact?(public_metadata)).to be(false)
expect(described_class.authorised_artifact?(admin_metadata)).to be(false)
end
described_class.set(user: create(:user, :admin)) do
expect(described_class.authorised_artifact?(admin_metadata)).to be(false)
end
described_class.set(user: admin) do
expect(described_class.authorised_artifact?(admin_metadata)).to be(true)
admin.update!(role: :guest)
expect(described_class.authorised_artifact?(admin_metadata)).to be(false)
end
end
it 'denies a blob attached to a discarded post including its derived attachment' do
post = create(:post)
blob = ActiveStorage::Blob.create_before_direct_upload!(filename: 'private.png',
byte_size: 1, checksum: Base64.strict_encode64('checksum'), content_type: 'image/png')
ActiveStorage::Attachment.create!(name: 'thumbnail', record: post, blob:)
PostVersion.create!(post:, version_no: 1, event_type: 'discard', url: post.url,
title: 'private', tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024))
variant = ActiveStorage::VariantRecord.create!(blob:, variation_digest: 'spec-variation')
derived = ActiveStorage::Blob.create_before_direct_upload!(filename: 'derived.png',
byte_size: 1, checksum: Base64.strict_encode64('checksum'), content_type: 'image/png')
ActiveStorage::Attachment.create!(name: 'image', record: variant, blob: derived)
described_class.set(user: nil) do
expect(described_class.blob_visible?(blob)).to be(false)
expect(described_class.blob_visible?(derived)).to be(false)
end
described_class.set(user: create(:user, :admin)) do
expect(described_class.blob_visible?(derived)).to be(true)
end
end
end
RSpec.describe 'Counts artifact policy and external references' do
it 'rejects files created before the masked-summary policy' do
HistoryAccess.set(user: nil) do
metadata = HistoryAccess.authority.except('artifact_policy')
expect(HistoryAccess.authorised_artifact?(metadata)).to be(false)
end
end
it 'does not expose a private linked tag ID through an external history snapshot' do
tag = create(:tag, primary_name: 'private-linked-name')
TagVersion.create!(tag:, version_no: 1, event_type: 'discard', name: 'private-linked-name',
category: 'general', aliases: '', parent_tag_ids: '', created_at: Time.utc(2024),
names: { ja: { name: 'private-linked-name', script_code: 'Jpan',
auto_generated: false, aliases: [] } })
external = create(:external_tag, name: 'public-external')
row = NicoTagVersion.create!(external_tag: external, version_no: 1, event_type: 'create',
linked_tag_ids: tag.id.to_s,
created_at: Time.utc(2024))
post = create(:post)
snapshot = PostVersion.create!(post:, version_no: 1, event_type: 'create',
title: 'external reference', url: post.url, tags: '',
tags_json: [{ external_tag_id: external.id }], parent_post_ids: '',
created_at: Time.utc(2024, 2))
HistoryAccess.set(user: nil) do
expect(HistoryAccess.visible(NicoTagVersion.where(id: row.id))).to be_empty
expect(HistoryAccess.visible(PostVersion.where(id: snapshot.id))).to be_empty
end
HistoryAccess.set(user: create(:user, :admin)) do
expect(HistoryAccess.visible(NicoTagVersion.where(id: row.id))).to exist
end
public_row = NicoTagVersion.create!(external_tag: external, version_no: 2, event_type: 'update',
linked_tag_ids: '',
created_at: Time.utc(2024, 7))
HistoryAccess.set(user: nil) do
expect(HistoryAccess.visible(NicoTagVersion.where(id: public_row.id))).to exist
expect(HistoryAccess.visible(PostVersion.where(id: snapshot.id))).to be_empty
end
end
it 'propagates the recorded tag version policy to its post snapshot' do
parent = create(:tag, primary_name: 'private parent')
child = create(:tag, primary_name: 'referencing child')
[[parent, 'discard', '', 'private parent'],
[child, 'create', parent.id.to_s, 'referencing child']].each do |tag, event, parents, name|
TagVersion.create!(tag:, version_no: 1, event_type: event, name:,
category: 'general', aliases: '', parent_tag_ids: parents, created_at: Time.utc(2023),
names: { ja: { name:, script_code: 'Jpan',
auto_generated: false, aliases: [] } })
end
post = create(:post)
row = PostVersion.create!(post:, version_no: 1, event_type: 'create', title: 'snapshot',
url: post.url, tags: '', parent_post_ids: '',
tags_json: [{ tag_id: child.id, version_no: 1, name: 'referencing child', sections: [] }],
created_at: Time.utc(2024))
HistoryAccess.set(user: nil) do
expect(HistoryAccess.visible(PostVersion.where(id: row.id))).to be_empty
end
end
end
+3 -4
ファイルの表示
@@ -10,23 +10,22 @@ RSpec.describe 'Shared Tag and ExternalTag IDs', type: :model do
expect(next_tag.id).to be > external.id
end
it 'does not reuse physically deleted IDs in either table' do
it 'does not reuse IDs after internal deletion while external tags remain' do
tag = create(:tag)
external = create(:external_tag)
TagName.where(tag_id: tag.id).delete_all
tag.destroy!
external.destroy!
expect(ExternalTag.exists?(external.id)).to be(true)
expect(create(:tag).id).to be > external.id
expect(create(:external_tag).id).to be > external.id
end
it 'retains existing overlapping IDs without renumbering on update or insert' do
it 'retains overlapping IDs after a tag update and a new allocation' do
tag = create(:tag)
external = create(:external_tag)
external.update_columns(id: tag.id)
tag.update!(category: :meta)
external.update!(name: 'updated_external')
before_tag = tag.reload.attributes
before_external = external.reload.attributes
+15 -5
ファイルの表示
@@ -47,24 +47,34 @@ RSpec.describe VersionRecord, type: :model do
external_tag: nico_tag,
version_no: 1,
event_type: 'create',
name: "nico:#{ nico_tag.name }",
linked_tags: '',
linked_tag_ids: '',
created_at: Time.current,
created_by_user: nil
)
expect {
version.update!(name: 'nico:changed')
version.update!(linked_tag_ids: '123')
}.to raise_error(ActiveRecord::ReadOnlyRecord)
end
it 'rejects discard and restore versions for external tags' do
[:discard, :restore].each do |event_type|
version = NicoTagVersion.new(
external_tag: nico_tag, version_no: 1, event_type:,
linked_tag_ids: '', created_at: Time.current)
expect(version).not_to be_valid
expect(version.errors[:event_type]).not_to be_empty
end
end
it 'prevents NicoTagVersion destroy' do
version = NicoTagVersion.create!(
external_tag: nico_tag,
version_no: 1,
event_type: 'create',
name: "nico:#{ nico_tag.name }",
linked_tags: '',
linked_tag_ids: '',
created_at: Time.current,
created_by_user: nil
)
+124
ファイルの表示
@@ -0,0 +1,124 @@
require 'rails_helper'
require 'tmpdir'
RSpec.describe 'Counts native GET acknowledgement', type: :request do
include ActiveSupport::Testing::TimeHelpers
around do |example|
Dir.mktmpdir('counts-download-request') do |path|
@counts_download_dir = Pathname.new(path)
example.run
end
end
before do
allow(CountsExportDownload).to receive(:directory).and_return(@counts_download_dir)
end
def ready_job(format, payload: 'payload')
CountsExportJobs.pulse
post '/counts/export/jobs', params: { resource: 'posts', definition: 'history',
mode: 'details', columns: 'resource_id', locale: 'ja', format: }
expect(response).to have_http_status(:accepted)
job = response.parsed_body.fetch('job')
signature = CountsExportJobs.verifier.verified(job, purpose: 'counts_export_job')
owner = signature.fetch('owner')
exporter = instance_double(CountsExporter, resource: 'posts', format:, body: [payload])
file = HistoryAccess.set(user: nil) { CountsExportDownload.prepare(exporter, owner) }
state = CountsExportJobs.read(signature.fetch('id'))
state.merge!('status' => 'ready', 'token' => file)
CountsExportJobs.write(state)
post "/counts/export/jobs/#{ job }/download"
expect(response).to have_http_status(:ok)
[job, file, response.parsed_body.fetch('download_attempt')]
end
['csv', 'json'].each do |format|
it "delivers every byte of a multi-chunk #{ format } file through HTTP" do
sign_in_as(create(:user, :member))
rows = Array.new(5000) { |index| { id: index, name: '日本語,名称' } }
payload = if format == 'json'
rows.to_json
else
"id,name\r\n" + rows.map { CSV.generate_line(_1.values) }.join
end
job, file, attempt = ready_job(format, payload:)
owner = CountsExportJobs.verifier.verified(job, purpose: 'counts_export_job').fetch('owner')
metadata = HistoryAccess.set(user: nil) { CountsExportDownload.find(file, owner) }
expected = File.binread(metadata.fetch('path'))
expect(expected.bytesize).to be > 64 * 1024
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:ok)
expect(response.body.bytesize).to eq(expected.bytesize)
expect(response.body.b).to eq(expected)
expect(response.body.b).to eq(payload.b)
get "/counts/export/jobs/#{ job }"
expect(response.parsed_body['download_status']).to eq('started')
end
it "acknowledges only an authorised #{ format } GET body, not HEAD or acceptance" do
job, file, attempt = ready_job(format)
get "/counts/export/jobs/#{ job }"
expect(response.parsed_body['download_status']).to eq('pending')
head "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:method_not_allowed)
get "/counts/export/jobs/#{ job }"
expect(response.parsed_body['download_status']).to eq('pending')
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:ok)
expect(response.headers['Content-Disposition']).to include("counts-posts.#{ format }")
expect(response.body).to eq('payload')
get "/counts/export/jobs/#{ job }"
expect(response.parsed_body['download_status']).to eq('started')
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:not_found)
end
end
it 'rejects wrong attempts, cancellation and a newly discarded resource' do
job, file, attempt = ready_job('json')
get "/counts/export/files/#{ file }", params: { job:, attempt: 'wrong' }
expect(response).to have_http_status(:not_found)
delete "/counts/export/jobs/#{ job }"
expect(response).to have_http_status(:ok)
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:not_found)
job, file, attempt = ready_job('json')
record = create(:post)
PostVersion.create!(post: record, version_no: 1, event_type: 'discard', title: 'private',
url: record.url, tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024))
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:not_found)
expect(response.body).not_to include('payload', 'private')
end
it 'does not let a second browser session inspect the job or download its file' do
job, file, attempt = ready_job('json')
other = ActionDispatch::Integration::Session.new(Rails.application)
other.get("/counts/export/jobs/#{ job }")
expect(other.response).to have_http_status(:not_found)
other.get("/counts/export/files/#{ file }", params: { job:, attempt: })
expect(other.response).to have_http_status(:not_found)
expect(other.response.body).not_to include('payload')
end
it 'does not acknowledge an expired attempt' do
job, file, attempt = ready_job('json')
travel_to(Time.current + 46.seconds) do
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:not_found)
get "/counts/export/jobs/#{ job }"
expect(response.parsed_body['download_status']).to eq('timeout')
end
end
it 'records a missing file without accepting the GET as started' do
job, file, attempt = ready_job('json')
owner = CountsExportJobs.verifier.verified(job, purpose: 'counts_export_job').fetch('owner')
metadata = HistoryAccess.set(user: nil) { CountsExportDownload.find(file, owner) }
File.unlink(metadata.fetch('path'))
get "/counts/export/files/#{ file }", params: { job:, attempt: }
expect(response).to have_http_status(:not_found)
get "/counts/export/jobs/#{ job }"
expect(response.parsed_body['download_status']).to eq('unavailable')
end
end
+40
ファイルの表示
@@ -0,0 +1,40 @@
require 'rails_helper'
RSpec.describe 'History boundaries shared with Counts', type: :request do
let!(:tag) { create(:tag, primary_name: 'description tag') }
let!(:description) do
TagDescription.create!(tag:, language_code: 'ja', version_no: 2, body: 'private body')
end
before do
['create', 'discard'].each_with_index do |event, index|
TagDescriptionVersion.create!(tag_id: tag.id, language_code: 'ja',
version_no: index + 1, event_type: event, body: 'private body',
created_at: Time.utc(2024))
end
end
it 'denies the same discarded description through Counts, the facade and the ordinary API' do
get '/counts/query', params: { resource: 'tag_descriptions', definition: 'history',
mode: 'details', locale: 'ja', execution_at: '2025-01-01T00:00:00Z' }
expect(response).to have_http_status(:ok)
expect(response.parsed_body).to include('count' => 0, 'output_count' => 2,
'excluded_count' => 2, 'versions' => [])
["/wiki/#{ description.id }", "/tags/#{ tag.id }/description"].each do |path|
get path, params: { locale: 'ja' }
expect(response).to have_http_status(:not_found)
expect(response.body).not_to include('private body')
end
get '/sitemap_entries/descriptions', params: { locale: 'ja' }
expect(response).to have_http_status(:ok)
expect(response.parsed_body['ids']).not_to include(description.id)
end
it 'permits the server-authenticated administrator to read the discarded description' do
sign_in_as(create(:user, :admin))
get "/tags/#{ tag.id }/description", params: { locale: 'ja' }
expect(response).to have_http_status(:ok)
expect(response.parsed_body['body']).to eq('private body')
expect(response.headers['Cache-Control']).to include('no-store')
end
end
+196
ファイルの表示
@@ -0,0 +1,196 @@
require 'rails_helper'
RSpec.describe 'Counts masked search', type: :request do
let!(:tag) { create(:tag, primary_name: '秘密のタグ') }
let(:actor) { create(:user, :member) }
let(:admin) { create(:user, :admin) }
let(:base) do
{ 'resource' => 'tags', 'definition' => 'history', 'mode' => 'summary',
'locale' => 'ja', 'execution_at' => '2025-01-01T00:00:00Z',
'timezone' => 'UTC', 'rows' => '', 'cols' => '', 'sort' => '' }
end
before do
[1, 2].each do |number|
TagVersion.create!(tag:, version_no: number,
event_type: number == 1 ? 'create' : 'discard', name: '秘密のタグ',
category: 'general', aliases: '秘密の別名', parent_tag_ids: '',
names: { ja: { name: '秘密のタグ', script_code: 'Jpan', auto_generated: false,
aliases: [{ name: '秘密の別名', script_code: 'Jpan' }] } },
created_by_user: actor, created_at: Time.utc(2023, number))
end
end
def query(extra = {})
get '/counts/query', params: base.merge(extra)
expect(response).to have_http_status(:ok), extra.inspect
response.parsed_body
end
def predicate(field, operator, value, index = 0)
prefix = "conditions.#{ index }"
result = { 'conditions' => '', "#{ prefix }.field" => field,
"#{ prefix }.operator" => operator }
if ['in', 'any', 'all', 'none', 'not_in'].include?(operator)
result.merge!("#{ prefix }.values" => '1', 'condition_values' => '',
"condition_values.#{ index }.condition" => index.to_s,
"condition_values.#{ index }.value_type" => 'string',
"condition_values.#{ index }.value" => value)
else
result["#{ prefix }.value"] = value
end
result
end
it 'rejects every name and alias predicate on private history' do
['name', 'name_ja', 'names.name', 'aliases.name', 'names.aliases.name'].each do |field|
secret = field.include?('aliases') ? '秘密の別名' : '秘密のタグ'
multiple = !['name', 'name_ja'].include?(field)
operators = multiple ? ['any', 'all', 'none'] : ['in', 'not_in']
['eq', 'ne', 'like', *operators].each do |operator|
[secret, '********', '%', ''].each do |value|
expected = (value == '' && ['eq', 'ne', 'like'].include?(operator)) ? 2 : 0
expect(query(predicate(field, operator, value))['count']).to eq(expected)
end
end
blank = predicate(field, 'eq', '').merge('conditions.0.blank' => '1')
expect(query(blank)['count']).to eq(0)
end
end
it 'ignores missing and empty scalar values unless blank is checked' do
['name', 'name_ja', 'names.name', 'aliases.name'].each do |field|
['eq', 'ne', 'like'].each do |operator|
condition = predicate(field, operator, '')
[condition, condition.except('conditions.0.value')].each do |empty|
expect(query(empty)['count']).to eq(2)
expect(query(empty.merge('conditions.0.blank' => '0'))['count']).to eq(2)
expect(query(empty.merge('conditions.0.blank' => '1'))['count']).to eq(0)
end
end
end
end
it 'keeps details totals unchanged for inactive private predicates' do
settings = predicate('name', 'ne', '').merge('mode' => 'details')
result = query(settings)
expect(result['count']).to eq(0)
expect(result['versions']).to be_empty
expect(result['output_count']).to eq(2)
active = query(settings.merge('conditions.0.blank' => '1'))
expect(active['output_count']).to eq(0)
end
it 'ignores missing sets but preserves explicit empty set semantics' do
{ 'name' => ['in', 'not_in'], 'aliases.name' => ['any', 'all', 'none'] }.each do
|field, operators|
operators.each do |operator|
condition = { 'conditions' => '', 'conditions.0.field' => field,
'conditions.0.operator' => operator }
expect(query(condition)['count']).to eq(2)
empty = condition.merge('conditions.0.values' => '1', 'condition_values' => '')
expected = ['in', 'any'].include?(operator) ? 0 : 2
expect(query(empty)['count']).to eq(expected)
expect(query(condition.merge('conditions.0.blank' => '1'))['count']).to eq(0)
# An explicit empty string element is a real value, not an unspecified set.
expect(query(predicate(field, operator, ''))['count']).to eq(0)
end
end
end
it 'ignores empty private predicates regardless of their position' do
first = predicate('name', 'ne', '')
.merge(predicate('resource_id', 'eq', tag.id.to_s, 1))
last = predicate('resource_id', 'eq', tag.id.to_s)
.merge(predicate('aliases.name', 'like', '', 1))
expect(query(first)['count']).to eq(2)
expect(query(last)['count']).to eq(2)
active = predicate('name', 'ne', 'unrelated')
.merge(predicate('aliases.name', 'like', '', 1))
expect(query(active)['count']).to eq(0)
end
it 'searches public literal markers and treats axis predicates as explicit' do
public_tag = create(:tag, primary_name: '********')
TagVersion.create!(tag: public_tag, version_no: 1, event_type: 'create',
name: '********', category: 'general', aliases: '', parent_tag_ids: '',
names: { ja: { name: '********', script_code: 'Jpan', auto_generated: false,
aliases: [] } }, created_at: Time.utc(2023))
expect(query['count']).to eq(3)
['eq', 'ne', 'like', 'in', 'not_in'].each do |operator|
expected = ['ne', 'not_in'].include?(operator) ? 0 : 1
expect(query(predicate('name', operator, '********'))['count']).to eq(expected)
end
blank = predicate('aliases.name', 'eq', '').merge('conditions.0.blank' => '1')
expect(query(blank)['count']).to eq(1)
['********', '', nil].each do |value|
condition = { 'conditions' => '', 'conditions.0.field' => 'name',
'conditions.0.operator' => 'axis_eq',
'conditions.0.value_type' => value.nil? ? 'null' : 'string' }
condition['conditions.0.value'] = value unless value.nil?
expected = value == '********' ? 1 : 0
expect(query(condition)['count']).to eq(expected)
drill = { 'drill' => '', 'drill.0.field' => 'name',
'drill.0.value_type' => value.nil? ? 'null' : 'string' }
drill['drill.0.value'] = value unless value.nil?
expect(query(drill)['count']).to eq(expected)
end
end
it 'keeps public identity and timeline predicates searchable' do
expect(query['count']).to eq(2)
values = { 'resource_id' => tag.id.to_s, 'version_no' => '2',
'event_type' => 'discard', 'user' => actor.id.to_s,
'created_at' => '2023-02-01T00:00:00Z' }
values.each do |field, value|
expected = ['resource_id', 'user'].include?(field) ? 2 : 1
expect(query(predicate(field, 'eq', value))['count']).to eq(expected)
end
condition = predicate('name', 'ne', '秘密のタグ')
.merge(predicate('event_type', 'eq', 'discard', 1))
expect(query(condition)['count']).to eq(0)
expect(query(predicate('category', 'eq', '********'))['count']).to eq(0)
expect(query(predicate('category', 'eq', 'general'))['count']).to eq(0)
end
it 'keeps public predicates available after combining multiple conditions' do
condition = predicate('resource_id', 'eq', tag.id.to_s)
.merge(predicate('event_type', 'eq', 'discard', 1))
expect(query(condition)['count']).to eq(1)
condition.merge!(predicate('name', 'ne', 'unrelated', 2))
expect(query(condition)['count']).to eq(0)
end
it 'keeps real name searches for administrators and excludes private details' do
condition = predicate('name', 'eq', '秘密のタグ')
details = query(condition.merge('mode' => 'details'))
expect(details['versions']).to be_empty
expect(details['output_count']).to eq(0)
masked = query(predicate('name', 'eq', '********').merge('mode' => 'details'))
expect(masked['versions']).to be_empty
expect(masked['output_count']).to eq(0)
sign_in_as(admin)
expect(query(condition)['count']).to eq(2)
expect(query(condition.merge('mode' => 'details'))['versions'].length).to eq(2)
end
it 'rejects private names reached through a post tag snapshot' do
post = create(:post)
PostVersion.create!(post:, version_no: 1, event_type: 'create', title: 'post',
url: post.url, tags: '', parent_post_ids: '',
tags_json: [{ tag_id: tag.id, version_no: 1, name: '秘密のタグ',
category: 'general', sections: [] }],
created_at: Time.utc(2024))
['tags.name', 'tags.names.name', 'tags.name_ja'].each do |field|
['eq', 'ne', 'like', 'any', 'all', 'none'].each do |operator|
['秘密のタグ', '********', '%'].each do |value|
settings = predicate(field, operator, value).merge('resource' => 'posts')
expect(query(settings)['count']).to eq(0)
end
end
blank = predicate(field, 'eq', '').merge('resource' => 'posts',
'conditions.0.blank' => '1')
expect(query(blank)['count']).to eq(0)
end
end
end
+11 -10
ファイルの表示
@@ -1,6 +1,6 @@
require 'rails_helper'
RSpec.describe 'Counts API', type: :request do
RSpec.describe 'Legacy Counts API', type: :request do
let!(:actor) { create(:user, :member) }
let!(:post) { create(:post) }
let!(:tag) { create(:tag, primary_name: 'counts tag', category: :general) }
@@ -28,16 +28,16 @@ RSpec.describe 'Counts API', type: :request do
context resource do
let(:path) { "/counts/#{ resource }" }
let(:id) { resource == 'posts' ? post.id : tag.id }
let(:filters) { { resource_id: id, user_id: actor.id,
let(:filters) { { resource_id: id, user_id: actor.id, latest: '0',
from: '2024-01-01T00:00Z', before: '2024-02-01T00:00Z' } }
it 'uses the same AND filters for SQL totals, groups and details' do
get path, params: filters.merge(axis: 'event_type')
get path, params: filters.merge(axis: 'month')
expect(response).to have_http_status(:ok)
summary = response.parsed_body
expect(summary['count']).to eq(2)
expect(summary['groups'].sum { _1['count'] }).to eq(2)
expect(summary['groups'].map { _1['key'] }).to eq(['create', 'update'])
expect(summary['groups'].map { _1['key'] }).to eq(['2024-01'])
get path, params: filters.merge(mode: 'details')
expect(response.parsed_body['count']).to eq(2)
@@ -45,16 +45,16 @@ RSpec.describe 'Counts API', type: :request do
end
it 'paginates details and groups independently from totals' do
get path, params: { resource_id: id, mode: 'details', limit: 1, page: 2 }
get path, params: { resource_id: id, latest: '0', mode: 'details', limit: 1, page: 2 }
expect(response.parsed_body['count']).to eq(3)
expect(response.parsed_body['versions'].sole['version_no']).to eq(2)
get path, params: { resource_id: id, axis: 'month', limit: 1, page: 2 }
get path, params: { resource_id: id, latest: '0', axis: 'month', limit: 1, page: 2 }
expect(response.parsed_body['group_count']).to eq(2)
expect(response.parsed_body['groups'].sole).to include('key' => '2024-02', 'count' => 1)
end
it 'groups null actors and supports their drill-down filter' do
get path, params: { resource_id: id, axis: 'user' }
get path, params: { resource_id: id, latest: '0', axis: 'user' }
expect(response.parsed_body['groups']).to include(
{ 'key' => nil, 'name' => nil, 'count' => 1 })
get path, params: { resource_id: id, mode: 'details', user_id: 'none' }
@@ -62,7 +62,7 @@ RSpec.describe 'Counts API', type: :request do
end
it 'uses an inclusive start and exclusive end' do
get path, params: { resource_id: id,
get path, params: { resource_id: id, latest: '0',
from: '2024-01-01T00:00Z', before: '2024-01-02T00:00Z' }
expect(response.parsed_body['count']).to eq(1)
end
@@ -88,9 +88,10 @@ RSpec.describe 'Counts API', type: :request do
end
it 'matches removed tags from the previous snapshot, not current post_tags' do
get '/counts/posts', params: { tag: tag.id, mode: 'details' }
get '/counts/posts', params: { tag: tag.id, latest: '0', mode: 'details' }
expect(response.parsed_body['count']).to eq(2)
expect(response.parsed_body['versions'].map { _1['version_no'] }).to eq([2, 1])
expect(response.parsed_body['versions'].first['tags'].sole['type']).to eq('removed')
expect(response.parsed_body['versions'].first['tags']).to be_empty
expect(response.parsed_body['versions'].last['tags'].sole['id']).to eq(tag.id)
end
end
+494
ファイルの表示
@@ -0,0 +1,494 @@
require 'rails_helper'
RSpec.describe 'Counts table API', type: :request do
include_context 'Ainu locale'
let!(:actor) { create(:user, :member, name: 'Counts actor') }
let!(:record) { create(:post) }
let!(:tag) { create(:tag, primary_name: 'alpha', category: :general) }
let!(:external) { create(:external_tag, name: 'beta') }
let(:point) { '2025-01-01T00:00:00Z' }
let(:base) do
{ resource: 'posts', definition: 'history', mode: 'details',
execution_at: point, timezone: 'UTC', locale: 'ja' }
end
before do
TagVersion.create!(tag:, version_no: 1, event_type: 'create', name: 'alpha',
category: 'general', aliases: '', parent_tag_ids: '',
names: { ja: { name: 'alpha', script_code: 'Jpan', auto_generated: false, aliases: [] } },
created_at: Time.utc(2023))
NicoTagVersion.create!(external_tag: external, version_no: 1,
event_type: 'create', linked_tag_ids: '', created_at: Time.utc(2023))
snapshots = [
{ tag_id: tag.id, version_no: 1, name: 'alpha', category: 'general', sections: [] },
{ external_tag_id: external.id }]
[1, 2, 3].each do |number|
PostVersion.create!(post: record, version_no: number,
event_type: number == 1 ? 'create' : 'update', url: record.url,
title: "snapshot #{ number }", tags: '', parent_post_ids: '',
tags_json: snapshots.first(number == 1 ? 2 : (number == 2 ? 1 : 0)),
created_by_user: number == 3 ? nil : actor,
created_at: Time.utc(2024, 1, 31, 15) + (number - 1).hours)
end
end
def query(extra = {})
get '/counts/query', params: base.merge(extra)
expect(response).to have_http_status(:ok)
response.parsed_body
end
def axis(field, value, type: 'string', grain: nil, root: 'conditions')
result = { root => '', "#{ root }.0.field" => field,
"#{ root }.0.value_type" => type }
result["#{ root }.0.operator"] = 'axis_eq' if root == 'conditions'
result["#{ root }.0.value"] = value unless type == 'null'
result["#{ root }.0.grain"] = grain if grain
result
end
def versions(result) = result.fetch('versions').map { _1.fetch('version_no') }
context 'nico link ID snapshots through the V1 field name' do
let(:base) do
super().merge(resource: 'nico_tags')
end
before do
NicoTagVersion.create!(external_tag: external, version_no: 2,
event_type: 'update', linked_tag_ids: tag.id.to_s,
created_at: Time.utc(2024, 1, 1))
NicoTagVersion.create!(external_tag: external, version_no: 3,
event_type: 'update', linked_tag_ids: '',
created_at: Time.utc(2024, 2, 1))
end
it 'returns ID values under linked_tags without exposing the physical column key' do
result = query
expect(versions(result)).to eq([1, 2, 3])
expect(result['versions'].map { _1.fetch('linked_tags') }).to eq(['', tag.id.to_s, ''])
result['versions'].each do |row|
expect(row).not_to have_key('linked_tag_ids')
expect(row.fetch('name')).to eq('nico:beta')
end
end
it 'preloads external names separately for aliased link-change details' do
statements = []
listener = lambda do |_name, _start, _finish, _id, payload|
statements << payload[:sql]
end
ActiveSupport::Notifications.subscribed(listener, 'sql.active_record') do
result = query(axis('changed_field', 'linked_tags'))
expect(versions(result)).to eq([2, 3])
expect(result.fetch('versions').map { _1.fetch('name') })
.to eq(['nico:beta', 'nico:beta'])
end
details = statements.find { _1.include?('AS counts_latest') }
expect(details).not_to be_nil
expect(details).not_to include('JOIN `external_tags`')
names = statements.select { _1.match?(/SELECT .* FROM `external_tags`/) }
expect(names.length).to eq(1)
end
it 'advertises the legacy field, label and default columns only' do
get '/counts/definitions/nico_tags', params: { locale: 'ja' }
expect(response).to have_http_status(:ok)
body = response.parsed_body
['fields', 'filter_fields', 'output_fields'].each do |collection|
keys = body.fetch(collection).map { _1.fetch('key') }
expect(keys).to include('linked_tags')
expect(keys).not_to include('linked_tag_ids')
end
field = body.fetch('output_fields').find { _1['key'] == 'linked_tags' }
expect(field.fetch('label')).to eq('counts:attributes.linked_tags')
body.fetch('axes').each_value do |keys|
expect(keys).to include('linked_tags')
expect(keys).not_to include('linked_tag_ids')
end
body.fetch('defaults').select { _1['mode'] == 'details' }.each do |entry|
settings = entry.fetch('params').to_h { [_1.fetch('key'), _1.fetch('value')] }
expect(settings.fetch('columns').split).to include('name', 'linked_tags')
expect(settings.fetch('columns').split).not_to include('linked_tag_ids')
end
end
it 'filters IDs and legacy drills and compares link changes through linked_tags' do
predicate = { conditions: '', 'conditions.0.field' => 'linked_tags',
'conditions.0.operator' => 'eq', 'conditions.0.value' => tag.id.to_s }
expect(versions(query(predicate))).to eq([2])
expect(versions(query(axis('linked_tags', tag.id.to_s, root: 'drill')))).to eq([2])
expect(versions(query(axis('changed_field', 'linked_tags')))).to eq([2, 3])
expect(versions(query(predicate.merge('conditions.0.value' => 'alpha')))).to be_empty
end
it 'sorts ID snapshots through the public field in both directions' do
expect(versions(query(sort: 'linked_tags:asc version_no:asc'))).to eq([1, 3, 2])
expect(versions(query(sort: 'linked_tags:desc version_no:asc'))).to eq([2, 1, 3])
end
it 'restores historical link IDs before applying a latest snapshot filter' do
predicate = { definition: 'latest', at: '2024-01-15T00:00:00Z',
conditions: '', 'conditions.0.field' => 'linked_tags',
'conditions.0.operator' => 'eq', 'conditions.0.value' => tag.id.to_s }
expect(versions(query(predicate))).to eq([2])
expect(versions(query(predicate.except(:at)))).to be_empty
end
it 'groups by linked_tags ID values and accepts the resulting drill condition' do
sign_in_as(create(:user, :admin))
result = query(mode: 'summary', rows: 'linked_tags', cols: '', sort: '')
expect(result.fetch('cells')).to match_array([
{ 'values' => [''], 'count' => 2 },
{ 'values' => [tag.id.to_s], 'count' => 1 }])
expect(versions(query(axis('linked_tags', tag.id.to_s)))).to eq([2])
end
it 'rejects the physical column as a public filter, axis, sort or change field' do
sign_in_as(create(:user, :admin))
[
axis('linked_tag_ids', tag.id.to_s),
axis('changed_field', 'linked_tag_ids'),
{ mode: 'summary', rows: 'linked_tag_ids', cols: '', sort: '' },
{ sort: 'linked_tag_ids:asc' }].each do |invalid|
get '/counts/query', params: base.merge(invalid)
expect(response).to have_http_status(:bad_request), invalid.inspect
end
end
end
it 'identifies a totals timeout without exposing SQL in the error response' do
allow_any_instance_of(CountsPivotTotals).to receive(:call)
.and_raise(CountsPivotTotals::TimedOut, 'private SQL')
get '/counts/query', params: base.merge(mode: 'summary', rows: 'created_at:month')
expect(response).to have_http_status(:service_unavailable)
expect(response.parsed_body).to eq('error' => 'counts_query_timeout',
'stage' => 'totals', 'timeout_seconds' => 30)
expect(response.body).not_to include('private SQL')
end
it 'applies the same optimizer safeguards to the paginated detail SELECT' do
statements = []
listener = lambda do |_name, _start, _finish, _id, payload|
statements << payload[:sql]
end
ActiveSupport::Notifications.subscribed(listener, 'sql.active_record') do
result = query(tags: 'alpha', limit: 1)
expect(result['count']).to eq(2)
expect(versions(result)).to eq([1])
end
details = statements.find { _1.include?('AS counts_latest') }
expect(details).not_to be_nil
expect(details).to include('MAX_EXECUTION_TIME(30000)',
"SET_VAR(optimizer_switch='semijoin=off')")
end
it 'separates latest and history and defaults to stable detail ordering and limit 100' do
history = query
expect(versions(history)).to eq([1, 2, 3])
expect(history).to include('limit' => 100, 'count' => 3, 'output_count' => 3,
'visible_output_count' => 3, 'excluded_count' => 0)
expect(history['versions'].map { _1['latest'] }).to eq([false, false, true])
expect(versions(query(definition: 'latest'))).to eq([3])
expect(versions(query(limit: 1, page: 2))).to eq([2])
expect(versions(query(sort: 'version_no:desc', limit: 1))).to eq([3])
end
it 'counts regular pivot rows separately from totals and from underlying records' do
total = query(mode: 'summary', rows: '', cols: '')
expect(total).to include('count' => 3, 'output_count' => 1, 'group_count' => 1)
expect(total['cells']).to eq([{ 'values' => [], 'count' => 3 }])
pivot = query(mode: 'summary', rows: 'event_type', cols: '', sort: '', limit: 1)
expect(pivot).to include('group_count' => 2, 'visible_output_count' => 2)
expect(pivot['row_headers'].count { _1['kind'] == 'data' }).to eq(1)
expect(pivot['row_headers']).to include(hash_including('kind' => 'grand_total'))
expect(query(mode: 'summary', rows: '', cols: 'event_type', sort: '')['output_count']).to eq(1)
end
it 'preserves timezone, NULL and legacy drill semantics and permits condition removal' do
month = axis('created_at', '2024-02', grain: 'month')
expect(versions(query(month.merge(timezone: 'Asia/Tokyo')))).to eq([1, 2, 3])
expect(versions(query(month))).to eq([])
old = axis('created_at', '2024-02', grain: 'month', root: 'drill')
expect(versions(query(old.merge(timezone: 'Asia/Tokyo')))).to eq([1, 2, 3])
expect(versions(query(old.merge(month).merge(timezone: 'Asia/Tokyo')))).to eq([1, 2, 3])
expect(versions(query(axis('user', nil, type: 'null')))).to eq([3])
expect(versions(query(axis('changed_field', 'title')))).to eq([2, 3])
expect(versions(query)).to eq([1, 2, 3])
end
it 'evaluates latest before filtering the version set' do
predicate = { conditions: '', 'conditions.0.field' => 'latest',
'conditions.0.operator' => 'eq', 'conditions.0.value' => 'false' }
expect(versions(query(predicate))).to eq([1, 2])
expect(versions(query(predicate.merge('conditions.0.value' => 'true')))).to eq([3])
end
it 'distinguishes the snapshot cutoff from the current latest flag and condition' do
past = { definition: 'latest', at: '2024-01-31T16:00:00Z' }
result = query(past)
expect(versions(result)).to eq([2])
expect(result['versions'].first['latest']).to be(false)
predicate = { conditions: '', 'conditions.0.field' => 'latest',
'conditions.0.operator' => 'eq', 'conditions.0.value' => 'false' }
expect(versions(query(past.merge(predicate)))).to eq([2])
expect(versions(query(past.merge(predicate).merge('conditions.0.value' => 'true'))))
.to eq([])
unchanged = query(definition: 'latest', at: '2024-02-01T00:00:00Z')
expect(versions(unchanged)).to eq([3])
expect(unchanged['versions'].first['latest']).to be(true)
expect(versions(query(sort: 'latest:desc resource_id:asc version_no:asc'))).to eq([3, 1, 2])
end
it 'checks the current latest flag beyond the execution timestamp' do
PostVersion.create!(post: record, version_no: 4, event_type: 'update',
url: record.url, title: 'after execution', tags: '', tags_json: [],
parent_post_ids: '', created_at: Time.utc(2025, 2, 1))
result = query
expect(versions(result)).to eq([1, 2, 3])
expect(result['versions'].map { _1['latest'] }).to eq([false, false, false])
snapshot = query(definition: 'latest')
expect(versions(snapshot)).to eq([3])
expect(snapshot['versions'].first['latest']).to be(false)
end
it 'keeps ANY, ALL and NONE distinct without multiplying parent records' do
values = { conditions: '', 'conditions.0.field' => 'tags',
'conditions.0.values' => '1', condition_values: '',
'condition_values.0.condition' => '0',
'condition_values.0.value_type' => 'string',
'condition_values.0.value' => "internal:#{ tag.id }",
'condition_values.1.condition' => '0',
'condition_values.1.value_type' => 'string',
'condition_values.1.value' => "external:#{ external.id }" }
{ 'any' => [1, 2], 'all' => [1], 'none' => [3] }.each do |operator, expected|
expect(versions(query(values.merge('conditions.0.operator' => operator)))).to eq(expected)
end
result = query(axis('tags', "internal:#{ tag.id }"))
expect(result['versions'].first['tags'].length).to eq(2)
end
it 'distinguishes IN, NOT IN, an empty selection and an unspecified selection' do
predicate = { conditions: '', 'conditions.0.field' => 'version_no',
'conditions.0.operator' => 'in', 'conditions.0.values' => '1',
condition_values: '', 'condition_values.0.condition' => '0',
'condition_values.0.value_type' => 'number',
'condition_values.0.value' => '1' }
expect(versions(query(predicate))).to eq([1])
expect(versions(query(predicate.merge('conditions.0.operator' => 'not_in')))).to eq([2, 3])
empty = predicate.reject { |key, _value| key.to_s.start_with?('condition_values.') }
expect(versions(query(empty))).to be_empty
expect(versions(query(empty.except('conditions.0.values')))).to eq([1, 2, 3])
end
it 'resolves current tag names for AND, OR and negative searches' do
expect(versions(query(tags: 'alpha'))).to eq([1, 2])
expect(versions(query(tags: 'alpha nico:beta'))).to eq([1])
expect(versions(query(tags: 'nico:beta alpha'))).to eq([1])
expect(versions(query(tags: 'alpha not:nico:beta'))).to eq([2])
expect(versions(query(tags: 'alpha nico:beta', match: 'any'))).to eq([1, 2])
expect(versions(query(tags: 'not:alpha'))).to eq([3])
end
context 'when current tag names differ from recorded snapshot names' do
before do
tag.tag_names.find_by!(language_code: 'ja', primary_flg: true)
.update!(name: 'renamed-alpha')
create(:tag_name, :alias, tag:, name: 'alpha-alias')
TagVersion.create!(tag:, version_no: 2, event_type: 'update', name: 'renamed-alpha',
category: 'general', aliases: 'alpha-alias', parent_tag_ids: '',
names: { ja: { name: 'renamed-alpha', script_code: 'Jpan', auto_generated: false,
aliases: [{ name: 'alpha-alias', script_code: 'Jpan' }] } },
created_at: Time.utc(2024, 2, 1))
PostVersion.create!(post: record, version_no: 4, event_type: 'update',
url: record.url, title: 'after rename', tags: '', parent_post_ids: '',
tags_json: [{ tag_id: tag.id, version_no: 2, name: 'renamed-alpha',
category: 'general', sections: [] }],
created_by_user: actor, created_at: Time.utc(2024, 2, 2))
end
it 'resolves current names and aliases to stable IDs in both definitions' do
{ 'history' => [1, 2, 4], 'latest' => [4] }.each do |definition, expected|
['renamed-alpha', 'alpha-alias'].each do |name|
expect(versions(query(tags: name, definition:))).to eq(expected)
end
end
result = query(tags: 'renamed-alpha')
expect(result['versions'].first['tags'].first['name']).to eq('alpha')
expect(versions(query(tags: 'renamed-alpha', definition: 'latest',
at: '2024-01-31T16:00:00Z'))).to eq([2])
expect(versions(query(tags: 'not:alpha-alias'))).to eq([3])
expect(versions(query(tags: 'renamed-alpha nico:beta'))).to eq([1])
expect(versions(query(tags: 'renamed-alpha nico:beta', match: 'any'))).to eq([1, 2, 4])
end
it 'does not match a historical name reassigned to another tag ID' do
create(:tag, primary_name: 'alpha')
expect(versions(query(tags: 'alpha'))).to eq([])
expect(versions(query(tags: 'unregistered-name'))).to eq([])
expect(versions(query(tags: 'not:unregistered-name'))).to eq([1, 2, 3, 4])
expect(versions(query(tags: 'renamed-alpha'))).to eq([1, 2, 4])
end
it 'resolves immutable external names across historical snapshots' do
expect(versions(query(tags: 'nico:beta'))).to eq([1])
expect(versions(query(tags: 'nico:renamed-beta'))).to eq([])
expect(versions(query(tags: 'not:nico:beta'))).to eq([2, 3, 4])
end
end
it 'rejects malformed structured conditions and axes without accepting arbitrary SQL' do
[
{ rows: '[{"field":"user"}]' }, { rows: 'user user' },
{ rows: 'user:month' }, { rows: 'password' }, { limit: 101 }, { limit: 0 },
{ page: '1x' }, { conditions: 'eyJmaWVsZCI6InVzZXIifQ==' },
{ conditions: '', 'conditions.1.field' => 'user' },
axis('password', '1'), axis('changed_field', 'title').merge(definition: 'latest'),
axis('user', 'yes', type: 'boolean'),
{ conditions: '', 'conditions.0.field' => 'title',
'conditions.0.operator' => 'eq', 'conditions.0.value' => 'x' * 8193 }].each do |invalid|
get '/counts/query', params: base.merge(invalid)
expect(response).to have_http_status(:bad_request), invalid.inspect
end
end
it 'rejects duplicate URL keys and JSON query payloads at the HTTP boundary' do
['resource=posts&tag_refs=', 'resource=posts&tag_refs.0.tag_id=1',
'resource=posts&limit=20&limit=50', 'resource=posts&headers=%5B%22Id.%22%5D',
'resource=posts&limit%5B%5D=20'].each do |query_string|
get "/counts/query?#{ query_string }"
expect(response).to have_http_status(:bad_request)
end
end
it 'reports column overflow separately from malformed input' do
101.times do |index|
PostVersion.create!(post: record, version_no: index + 4,
event_type: 'update', url: record.url, title: "column #{ index }",
tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024, 3))
end
get '/counts/query', params: base.merge(mode: 'summary', rows: '', cols: 'title', sort: '')
expect(response).to have_http_status(:unprocessable_entity)
expect(response.parsed_body).to eq('error' => 'too_many_columns', 'max_columns' => 100)
end
it 'defines all six resources with filterable output fields and shared detail defaults' do
locale
CountsResourceDefinition::DEFINITIONS.each_key do |resource|
get "/counts/definitions/#{ resource }", params: { locale: 'ain-Kana' }
expect(response).to have_http_status(:ok)
body = response.parsed_body
filters = body['filter_fields'].map { _1['key'] }
walk = lambda do |fields, prefix|
fields.each do |field|
key = [prefix, field['key']].compact.join('.')
expect(filters).to include(key)
walk.call(field.fetch('children', []), key)
end
end
walk.call(body['output_fields'], nil)
defaults = body['defaults'].select { _1['mode'] == 'details' }
expect(defaults.map { _1['definition'] }).to eq(['latest', 'history'])
expect(defaults.first['params']).to eq(defaults.last['params'])
settings = defaults.first['params'].to_h { [_1['key'], _1['value']] }
expect(settings['sort']).to eq('resource_id:asc version_no:asc')
if resource == 'posts'
expect(settings['children'].split).to eq(['tags.name_ain', 'parents.id'])
end
keys = body['name_columns'].map { _1['key'] }
languages = Language.order(:created_at, :code).pluck(:code)
expect(keys).to eq(languages.map { "name_#{ _1 }" } + languages.map { "aliases_#{ _1 }" })
end
end
[nil, :guest, :member, :admin].each do |role|
context "authority #{ role || 'anonymous' }" do
before do
sign_in_as(create(:user, role)) if role
# Equal timestamps deliberately require version ordering, not a date comparison.
['discard', 'restore', 'discard', 'restore'].each_with_index do |event, index|
PostVersion.create!(post: record, version_no: index + 4, event_type: event,
url: record.url, title: event == 'restore' ? 'restored' : 'private snapshot',
tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024, 3))
end
end
it 'keeps every earlier discard boundary private but counts denied records' do
result = query
expected = role == :admin ? (1..7).to_a : [7]
expect(versions(result)).to eq(expected)
expect(result).to include('output_count' => 7,
'visible_output_count' => expected.length, 'excluded_count' => 7 - expected.length)
expect(response.headers['Cache-Control']).to include('no-store')
expect(response.body).not_to include('private snapshot') unless role == :admin
end
it 'applies the same version boundary to the ordinary history endpoint and old values' do
get '/posts/versions', params: { post: record.id, locale: 'ja' }
expect(response).to have_http_status(:ok)
rows = response.parsed_body.fetch('versions')
expect(rows.map { _1['version_no'] }).to eq(role == :admin ? (1..7).to_a.reverse : [7])
unless role == :admin
expect(rows.sole.dig('title', 'prev')).to be_nil
expect(response.body).not_to include('private snapshot')
end
end
it 'does not resolve private condition names even when the result set is empty' do
TagVersion.create!(tag:, version_no: 2, event_type: 'discard', name: 'secret tag',
category: 'general', aliases: '', parent_tag_ids: '',
names: { ja: { name: 'secret tag', script_code: 'Jpan', auto_generated: false, aliases: [] } },
created_at: Time.utc(2024, 2))
result = query(axis('tags', "internal:#{ tag.id }").merge(definition: 'latest'))
expect(result['versions']).to be_empty
expected = role == :admin ? { "internal:#{ tag.id }" => 'secret tag' } : {}
expect(result['condition_labels']['tags']).to eq(expected)
get '/counts/definitions/posts/candidates', params: {
field: 'tags', query: 'secret', execution_at: point, locale: 'ja' }
expect(response).to have_http_status(:ok)
expect(response.parsed_body['rows']).to be_empty
end
end
end
it 'includes private history in the public timeline without an exclusion count' do
PostVersion.create!(post: record, version_no: 4, event_type: 'discard', url: record.url,
title: 'hidden title', tags: '', tags_json: [], parent_post_ids: '',
created_at: Time.utc(2024, 3))
result = query(mode: 'summary', rows: 'created_at:month', cols: 'event_type')
expect(result['excluded_count']).to be_nil
expect(result['count']).to eq(4)
expect(result['cells'].sum { _1['count'] }).to eq(4)
expect(response.body).not_to include('hidden title')
masked = query(mode: 'summary', rows: 'resource_id', cols: '')
expect(masked['cells'].map { _1['values'] }).to eq([[record.id]])
expect(masked['cells'].first['count']).to eq(4)
expect(query['versions']).to be_empty
end
it 'allows public identity predicates without exposing private snapshot fields' do
PostVersion.create!(post: record, version_no: 4, event_type: 'discard',
url: record.url, title: 'hidden title', tags: '', tags_json: [],
parent_post_ids: '', created_at: Time.utc(2024, 3))
get '/counts/query', params: base.merge(mode: 'summary',
resource_id: record.id.to_s, rows: 'resource_id', cols: 'title')
expect(response).to have_http_status(:ok)
expect(json['count']).to eq(4)
expect(json['cells'].sum { _1['count'] }).to eq(4)
expect(json['cells'].map { _1['values'].last })
.to include(CountsAxes::MASKED_VALUE)
expect(response.body).not_to include('hidden title')
end
it 'reports the combined axis limit without replacing it with a generic 400' do
get '/counts/query', params: base.merge(mode: 'summary', sort: '',
rows: 'resource_id version_no event_type user created_at:month',
cols: 'title url thumbnail_base video_ms')
expect(response).to have_http_status(:bad_request)
expect(response.parsed_body).to eq('error' => 'too_many_axes', 'max_axes' => 8)
end
end
+8 -14
ファイルの表示
@@ -38,12 +38,9 @@ RSpec.describe 'NicoTags', type: :request do
end
it 'filters by nico tag name, linked tag name, and link status' do
linked = create(:external_tag)
linked.update!(name: 'search_linked')
unlinked = create(:external_tag)
unlinked.update!(name: 'search_unlinked')
other = create(:external_tag)
other.update!(name: 'other')
linked = create(:external_tag, name: 'search_linked')
unlinked = create(:external_tag, name: 'search_unlinked')
other = create(:external_tag, name: 'other')
destination = create(:tag, category: :general)
primary_tag_name_for(destination, 'ja').update!(name: 'destination_search')
NicoTagRelation.create!(nico_tag: linked, tag: destination)
@@ -80,11 +77,9 @@ RSpec.describe 'NicoTags', type: :request do
end
it 'sorts by name and timestamps' do
older = create(:external_tag)
older.update!(name: 'ordered_a')
older = create(:external_tag, name: 'ordered_a')
older.update_columns(created_at: 2.days.ago)
newer = create(:external_tag)
newer.update!(name: 'ordered_b')
newer = create(:external_tag, name: 'ordered_b')
newer.update_columns(created_at: 1.day.ago)
older_post_tag =
PostExternalTag.create!(post: create(:post), external_tag: older)
@@ -189,9 +184,8 @@ RSpec.describe 'NicoTags', type: :request do
versions = nico_tag.reload.nico_tag_versions.order(:version_no)
expect(versions.map(&:event_type)).to eq(['create', 'update'])
expect(versions.last.linked_tags.split).to match_array([
'nico_linked_a',
'nico_linked_b'])
expect(versions.last.linked_tag_ids.split.map(&:to_i))
.to match_array([linked_a.id, linked_b.id])
expect(versions.last.created_by_user_id).to eq(admin.id)
end
@@ -210,7 +204,7 @@ RSpec.describe 'NicoTags', type: :request do
expect(json).to eq([])
expect(nico_tag.reload.linked_tags).to be_empty
expect(nico_tag.nico_tag_versions.order(:version_no).last)
.to have_attributes(linked_tags: '', created_by_user: member)
.to have_attributes(linked_tag_ids: '', created_by_user: member)
end
it 'returns the tags field error when a nico tag is specified directly' do
+129
ファイルの表示
@@ -0,0 +1,129 @@
require 'rails_helper'
RSpec.describe 'Post history privacy', type: :request do
let(:origin) { Time.utc(2020, 1, 1) }
let!(:post_record) do
create(:post, title: 'current', created_at: origin, updated_at: origin)
end
let!(:other_post) do
create(:post, created_at: origin, updated_at: origin + 3.days + 1.hour)
end
let!(:tag) { create(:tag, primary_name: 'private-history-tag') }
let!(:external) { create(:external_tag, name: 'external-history-tag') }
before do
['create', 'discard'].each_with_index do |event, index|
TagVersion.create!(tag:, version_no: index + 1, event_type: event,
name: 'private-history-tag', category: 'general', aliases: '', parent_tag_ids: '',
names: { ja: { name: 'private-history-tag', script_code: 'Jpan',
auto_generated: false, aliases: [] } },
created_at: origin + index.hours)
end
NicoTagVersion.create!(external_tag: external, version_no: 1, event_type: 'create',
linked_tag_ids: tag.id.to_s,
created_at: origin)
['create', 'discard', 'restore', 'update'].each_with_index do |event, index|
snapshots =
if index < 2
[{ tag_id: tag.id, version_no: 1, name: 'private-history-tag', sections: [] }]
elsif index == 3
[{ external_tag_id: external.id }]
else
[]
end
PostVersion.create!(post: post_record, version_no: index + 1, event_type: event,
title: "version #{ index + 1 }", url: post_record.url, tags: '',
tags_json: snapshots, parent_post_ids: '', created_at: origin + (index + 1).days)
end
end
it 'evaluates each reference condition against the restored snapshot alone' do
HistoryAccess.set(user: nil) do
restored = PostVersion.where(post_id: post_record.id, version_no: 3)
HistoryAccess.reference_sql('post_versions', 'post_versions').each do |condition|
expect(restored.where(Arel.sql(condition)).pluck(:version_no)).to eq([3])
end
expect(HistoryAccess.visible(PostVersion.where(post_id: post_record.id))
.pluck(:version_no)).to eq([3])
end
end
it 'keeps every version through a later discard private after another restore' do
['discard', 'restore'].each_with_index do |event, index|
PostVersion.create!(post: post_record, version_no: index + 5, event_type: event,
title: 'later snapshot', url: post_record.url, tags: '', tags_json: [],
parent_post_ids: '', created_at: origin + (index + 5).days)
end
get '/posts/versions', params: { post: post_record.id }
expect(response).to have_http_status(:ok)
expect(response.parsed_body.fetch('versions').pluck('version_no')).to eq([6])
expect(response.parsed_body.fetch('count')).to eq(1)
expect(response.parsed_body.fetch('versions').first.fetch('title').fetch('prev')).to be_nil
end
['anonymous', 'member', 'admin'].each do |role|
context "as #{ role }" do
before do
sign_in_as(create(:user, role.to_sym)) unless role == 'anonymous'
end
it 'uses only authorised history timestamps for display' do
get '/posts'
expect(response).to have_http_status(:ok)
row = response.parsed_body.fetch('posts').find { _1.fetch('id') == post_record.id }
expected = origin + (role == 'admin' ? 4 : 3).days
expect(Time.zone.parse(row.fetch('updated_at'))).to eq(expected)
end
it 'uses the same visibility for date bounds and count' do
get '/posts', params: { updated_from: (origin + 4.days).iso8601,
updated_to: (origin + 4.days).iso8601 }
expect(response).to have_http_status(:ok)
ids = response.parsed_body.fetch('posts').pluck('id')
expect(ids).to eq(role == 'admin' ? [post_record.id] : [])
expect(response.parsed_body.fetch('count')).to eq(ids.length)
end
it 'sorts using authorised timestamps in both directions' do
['asc', 'desc'].each do |direction|
get '/posts', params: { order: "updated_at:#{ direction }" }
expect(response).to have_http_status(:ok)
expected = role == 'admin' ? [other_post.id, post_record.id] :
[post_record.id, other_post.id]
expected.reverse! if direction == 'desc'
expect(response.parsed_body.fetch('posts').pluck('id')).to eq(expected)
end
end
it 'applies authorised timestamp ordering before pagination' do
get '/posts', params: { order: 'updated_at:desc', limit: 1 }
expect(response).to have_http_status(:ok)
expected = role == 'admin' ? post_record.id : other_post.id
expect(response.parsed_body.fetch('posts').pluck('id')).to eq([expected])
expect(response.parsed_body.fetch('count')).to eq(2)
end
it 'counts only visible versions and masks a private previous version' do
get '/posts/versions', params: { post: post_record.id }
expect(response).to have_http_status(:ok)
body = response.parsed_body
expected = role == 'admin' ? [4, 3, 2, 1] : [3]
expect(body.fetch('count')).to eq(expected.length)
expect(body.fetch('versions').pluck('version_no')).to eq(expected)
restored = body.fetch('versions').find { _1.fetch('version_no') == 3 }
expect(restored.fetch('title').fetch('prev')).to eq(role == 'admin' ? 'version 2' : nil)
end
it 'uses only an authorised previous version for tag filtering' do
get '/posts/versions', params: { post: post_record.id, tag: tag.id }
expect(response).to have_http_status(:ok)
expected = role == 'admin' ? [3, 2, 1] : []
expect(response.parsed_body.fetch('versions').pluck('version_no')).to eq(expected)
expect(response.parsed_body.fetch('count')).to eq(expected.length)
end
end
end
end
+26 -11
ファイルの表示
@@ -11,34 +11,49 @@ RSpec.describe 'Post filter identity across locales', type: :request do
let!(:other_post) { create(:post) }
before do
# Legacy data can predate the current prohibition on internal nico: names.
primary_tag_name_for(tag, 'ja').update_columns(name: 'nico:shared')
primary_tag_name_for(tag, 'ain').update!(name: 'シロシ', auto_generated: false)
create(:tag_name, :alias, tag:, name: 'union_alias', language_code: 'ja')
PostTag.create!(post: internal_post, tag:)
PostExternalTag.create!(post: external_post, external_tag: external)
end
['nico:shared', 'union_alias', 'not:nico:shared'].each do |literal|
literals = ['legacy_union_source', 'union_alias', 'not:legacy_union_source', 'nico:shared']
literals.each do |literal|
it "preserves the complete match set for #{ literal }" do
get '/posts', params: { locale: 'ja', tags: literal }
expect(response).to have_http_status(:ok)
original_ids = json.fetch('posts').map { _1.fetch('id') }
expected = literal.start_with?('not:') ? [other_post.id] :
[internal_post.id, external_post.id]
expected = if literal.start_with?('not:')
[external_post.id, other_post.id]
elsif literal == 'nico:shared'
[external_post.id]
else
[internal_post.id]
end
expect(original_ids).to match_array(expected)
get '/posts/tag_filters', params: {
locale: 'ja', target_locale: 'ain-Kana', tags: literal }
expect(response).to have_http_status(:ok)
references = json.fetch('references')
expect(references.first).to include('tag_id' => tag.id, 'external_tag_id' => external.id)
get '/posts', params: {
locale: 'ain-Kana', tags: references.map { _1.fetch('name') }.join(' '),
tag_refs: references.to_json }
translated = json.fetch('tags')
expect(translated).to eq(literal == 'nico:shared' ? literal :
(literal.start_with?('not:') ? 'not:シロシ' : 'シロシ'))
get '/posts', params: { locale: 'ain-Kana', tags: translated }
expect(response).to have_http_status(:ok)
expect(json.fetch('posts').map { _1.fetch('id') }).to match_array(original_ids)
end
end
it 'rejects a locale conversion that cannot preserve a legacy union' do
primary_tag_name_for(tag, 'ja').update_columns(name: 'nico:shared')
get '/posts/tag_filters', params: {
locale: 'ja', target_locale: 'ain-Kana', tags: 'nico:shared' }
expect(response).to have_http_status(:unprocessable_entity)
end
it 'rejects removed reference parameters instead of overriding the name' do
get '/posts', params: { locale: 'ja', tags: 'legacy_union_source',
tag_refs: '', 'tag_refs.0.tag_id' => external.id.to_s }
expect(response).to have_http_status(:bad_request)
end
end
+41
ファイルの表示
@@ -629,6 +629,37 @@ RSpec.describe 'Posts API', type: :request do
create(:post_tag, post: plain_post, tag:, created_at: t3)
end
it 'displays history time after pagination without joining history to posts or count' do
queries = []
subscriber = ->(event) { queries << event.payload.fetch(:sql) }
ActiveSupport::Notifications.subscribed(subscriber, 'sql.active_record') do
get '/posts', params: { url: '/version-time/', order: 'url:asc', limit: 1 }
end
expect(response).to have_http_status(:ok)
expect(json.fetch('posts').pluck('id')).to eq([history_post.id])
expect(Time.zone.parse(json.fetch('posts').first.fetch('updated_at'))).to eq(t2)
expect(json.fetch('count')).to eq(3)
post_queries = queries.select { _1.include?('FROM `posts`') }
expect(post_queries).not_to be_empty
expect(post_queries.grep(/post_versions/)).to be_empty
end
it 'omits the history aggregate from count when only sorting by update time' do
queries = []
subscriber = ->(event) { queries << event.payload.fetch(:sql) }
ActiveSupport::Notifications.subscribed(subscriber, 'sql.active_record') do
get '/posts', params: { url: '/version-time/', order: 'updated_at:desc', limit: 1 }
end
expect(response).to have_http_status(:ok)
expect(json.fetch('posts').pluck('id')).to eq([newer_post.id])
expect(json.fetch('count')).to eq(3)
count_queries = queries.select { _1.include?('FROM `posts`') && _1.match?(/COUNT\(/i) }
expect(count_queries).not_to be_empty
expect(count_queries.grep(/post_versions/)).to be_empty
end
['asc', 'desc'].each do |direction|
it "sorts by the later of post update and latest version time (#{ direction })" do
get '/posts', params: { url: '/version-time/', order: "updated_at:#{ direction }" }
@@ -658,6 +689,16 @@ RSpec.describe 'Posts API', type: :request do
expect(json.fetch('posts').map { |item| item.fetch('id') }).to eq([history_post.id])
expect(json.fetch('count')).to eq(1)
end
it 'counts every timestamp match before pagination' do
get '/posts', params: { url: '/version-time/', order: 'updated_at:asc',
updated_from: t1.iso8601, updated_to: t3.iso8601,
limit: 1, page: 2 }
expect(response).to have_http_status(:ok)
expect(json.fetch('posts').pluck('id')).to eq([history_post.id])
expect(json.fetch('count')).to eq(3)
end
end
context 'when original_created_from/original_created_to are provided' do
+290
ファイルの表示
@@ -0,0 +1,290 @@
require 'rails_helper'
require 'tmpdir'
RSpec.describe 'Static Mirror API', type: :request do
include ActiveSupport::Testing::TimeHelpers
around do |example|
Dir.mktmpdir('static-mirror-spec') do |path|
allow(StaticMirrorJob).to receive(:directory).and_return(Pathname.new(path))
example.run
end
end
before do
allow(Rails.env).to receive(:development?).and_return(true)
allow(StaticMirrorJob).to receive(:perform_later)
end
def create_job
post '/static_mirror/jobs', params: { locale: 'ja' }, as: :json
expect(response).to have_http_status(:accepted)
json.fetch('job')
end
it 'accepts anonymous jobs asynchronously and serves the completed ZIP' do
record = create(:post, title: 'mirror-public-api-title')
PostVersion.create!(post: record, version_no: 1, event_type: 'create',
title: record.title, url: record.url, tags: '', tags_json: [], parent_post_ids: '',
created_at: Time.current)
get '/static_mirror'
expect(json).to include('resource_limit' => 100, 'concurrency' => 1)
id = create_job
expect(StaticMirrorJob).to have_received(:perform_later).with(id)
expect(StaticMirrorJob.directory.join('archive.zip')).not_to exist
get "/static_mirror/jobs/#{ id }"
expect(json).to include('status' => 'queued', 'data_at' => nil)
StaticMirrorJob.perform_now(id)
get "/static_mirror/jobs/#{ id }"
expect(json).to include('status' => 'ready', 'processed' => 1, 'total' => 1)
expect(json.fetch('data_at')).to be_present
expect(json.keys).not_to include('owner', 'origin', 'actor_id', 'policy_revision',
'publication_policy', 'history_revision', 'post_ids', 'tag_ids', 'content_digest')
get "/static_mirror/files/#{ id }"
expect(response).to have_http_status(:ok)
expect(response.media_type).to eq('application/zip')
expect(response.body.byteslice(0, 4)).to eq("PK\x03\x04".b)
expect(response.body).to include(record.title)
expect(response.headers['Cache-Control']).to include('no-store')
end
it 'bakes the canonical source origin regardless of request and forwarded hosts' do
host! 'mirror-intake.test'
post '/static_mirror/jobs', params: { locale: 'ja' }, as: :json,
headers: { 'X-Forwarded-Host' => 'untrusted-source.test',
'X-Forwarded-Proto' => 'https' }
expect(response).to have_http_status(:accepted)
id = json.fetch('job')
expect(StaticMirrorJob.state).not_to have_key('origin')
StaticMirrorJob.perform_now(id)
get "/static_mirror/files/#{ id }"
expect(response).to have_http_status(:ok)
expect(response.body).to include('https://hub.nizika.monster/ja')
expect(response.body).not_to include('mirror-intake.test', 'untrusted-source.test')
end
it 'rejects historical and unknown input instead of pretending to restore it' do
['at', 'data_at', 'admin', 'limit'].each do |key|
post '/static_mirror/jobs', params: { locale: 'ja', key => '2020-01-01' }, as: :json
expect(response).to have_http_status(:bad_request)
end
expect(StaticMirrorJob).not_to have_received(:perform_later)
end
it 'limits requests globally even across fresh anonymous sessions' do
create_job
other = ActionDispatch::Integration::Session.new(Rails.application)
other.post '/static_mirror/jobs', params: { locale: 'ja' }, as: :json
expect(other.response).to have_http_status(:too_many_requests)
expect(other.response.headers['Retry-After']).to eq('60')
expect(StaticMirrorJob).to have_received(:perform_later).once
end
it 'binds job status and downloads to the creating session' do
id = create_job
StaticMirrorJob.perform_now(id)
other = ActionDispatch::Integration::Session.new(Rails.application)
["/static_mirror/jobs/#{ id }", "/static_mirror/files/#{ id }"].each do |path|
other.get path
expect(other.response).to have_http_status(:not_found)
end
end
it 'refuses ready artifacts after a new discard even for admins' do
id = create_job
StaticMirrorJob.perform_now(id)
record = create(:post)
PostVersion.create!(post: record, version_no: 1, event_type: 'discard',
url: record.url, tags: '', tags_json: [], parent_post_ids: '', created_at: Time.current)
sign_in_as(create(:user, :admin))
get "/static_mirror/files/#{ id }"
expect(response).to have_http_status(:not_found)
get "/static_mirror/jobs/#{ id }"
expect(response).to have_http_status(:not_found)
end
it 'rechecks ready ZIPs after a non-discard content change' do
record = create(:post, title: 'mirror-withdrawn-title')
PostVersion.create!(post: record, version_no: 1, event_type: 'create',
title: record.title, url: record.url, tags: '', tags_json: [], parent_post_ids: '',
created_at: Time.current)
id = create_job
StaticMirrorJob.perform_now(id)
record.update!(title: 'mirror-corrected-title')
get "/static_mirror/jobs/#{ id }"
expect(response).to have_http_status(:not_found)
get "/static_mirror/files/#{ id }"
expect(response).to have_http_status(:not_found)
expect(response.body).not_to include('mirror-withdrawn-title')
end
it 'fails lost jobs and rejects expired downloads' do
id = create_job
travel 80.seconds do
get "/static_mirror/jobs/#{ id }"
expect(json).to include('status' => 'failed', 'error' => 'worker_stopped')
end
travel 11.minutes do
get "/static_mirror/files/#{ id }"
expect(response).to have_http_status(:not_found)
end
end
it 'rejects a previously ready ZIP after its expiry' do
id = create_job
StaticMirrorJob.perform_now(id)
travel 11.minutes do
get "/static_mirror/files/#{ id }"
expect(response).to have_http_status(:not_found)
get "/static_mirror/jobs/#{ id }"
expect(response).to have_http_status(:not_found)
end
end
it 'preserves the queued reservation after the cooldown' do
id = create_job
travel 61.seconds do
post '/static_mirror/jobs', params: { locale: 'ja' }, as: :json
expect(response).to have_http_status(:too_many_requests)
expect(StaticMirrorJob.state).to include('id' => id, 'status' => 'queued')
expect(StaticMirrorJob).to have_received(:perform_later).once
StaticMirrorJob.perform_now(id)
get "/static_mirror/jobs/#{ id }"
expect(json).to include('status' => 'failed', 'error' => 'generation_timeout')
end
end
it 'reclaims a stalled queued reservation without waiting for a status request' do
id = create_job
travel 80.seconds do
second_id = create_job
expect(second_id).not_to eq(id)
StaticMirrorJob.perform_now(id)
expect(StaticMirrorJob.state).to include('id' => second_id, 'status' => 'queued')
StaticMirrorJob.perform_now(second_id)
get "/static_mirror/jobs/#{ second_id }"
expect(json).to include('status' => 'ready')
end
end
it 'waits for brief intake lock contention instead of losing the queued worker' do
id = create_job
path = StaticMirrorJob.directory.join('.generation.lock')
lock = File.open(path, File::RDWR)
lock.flock(File::LOCK_EX)
attempts = Queue.new
worker = StaticMirrorJob.new
allow(worker).to receive(:generate)
allow(File).to receive(:open).and_call_original
allow(File).to receive(:open).with(path, File::RDWR).and_wrap_original do |original, *args, &block|
original.call(*args) do |file|
allow(file).to receive(:flock).and_wrap_original do |flock, mode|
attempts << mode
flock.call(mode)
end
block.call(file)
end
end
thread = Thread.new { worker.perform(id) }
Timeout.timeout(5) do
expect(attempts.pop).to eq(File::LOCK_EX)
lock.close
thread.join
end
expect(worker).to have_received(:generate).with(id, hash_including('status' => 'queued'))
ensure
lock&.close unless lock&.closed?
thread&.kill
thread&.join
end
it 'keeps the generation lock reserved even when stale metadata expires' do
create_job
lock = File.open(StaticMirrorJob.directory.join('.generation.lock'), File::RDWR)
lock.flock(File::LOCK_EX)
travel 80.seconds do
post '/static_mirror/jobs', params: { locale: 'ja' }, as: :json
expect(response).to have_http_status(:too_many_requests)
end
ensure
lock&.close
end
it 'preserves expiry between reading the queue and starting generation' do
id = create_job
worker = StaticMirrorJob.new
allow(worker).to receive(:generate).and_wrap_original do |original, *arguments|
travel 80.seconds do
get "/static_mirror/jobs/#{ id }"
expect(json).to include('status' => 'failed', 'error' => 'worker_stopped')
original.call(*arguments)
end
end
worker.perform(id)
expect(StaticMirrorJob.state).to include('status' => 'failed', 'error' => 'worker_stopped')
end
it 'rejects late progress and completion after a running job expires' do
id = create_job
mirror = instance_double(StaticMirror, data_at: Time.current.utc.iso8601(6), authority: { })
allow(StaticMirror).to receive(:new).and_return(mirror)
allow(mirror).to receive(:export) do |&progress|
travel 80.seconds do
get "/static_mirror/jobs/#{ id }"
expect(json).to include('status' => 'failed', 'error' => 'worker_stopped')
progress.call(1, 1)
end
'late archive bytes'
end
StaticMirrorJob.perform_now(id)
expect(StaticMirrorJob.state).to include('status' => 'failed', 'error' => 'worker_stopped',
'processed' => 0, 'total' => nil)
get "/static_mirror/files/#{ id }"
expect(response).to have_http_status(:not_found)
end
it 'reports bounded-generation failures without returning internal exception messages' do
id = create_job
allow(StaticMirror).to receive(:new).and_raise(StaticMirror::TooLarge, 'private details')
StaticMirrorJob.perform_now(id)
get "/static_mirror/jobs/#{ id }"
expect(json).to include('status' => 'failed', 'error' => 'too_large')
expect(response.body).not_to include('private details')
get "/static_mirror/files/#{ id }"
expect(response).to have_http_status(:not_found)
end
it 'is unavailable outside development' do
allow(Rails.env).to receive(:development?).and_return(false)
get '/static_mirror'
expect(response).to have_http_status(:not_found)
post '/static_mirror/jobs', params: { locale: 'ja' }, as: :json
expect(response).to have_http_status(:not_found)
get '/static_mirror/jobs/unknown'
expect(response).to have_http_status(:not_found)
get '/static_mirror/files/unknown'
expect(response).to have_http_status(:not_found)
end
it 'preserves user and IP BAN checks' do
user = create(:user, banned_at: Time.current)
sign_in_as(user)
post '/static_mirror/jobs', params: { locale: 'ja' }, as: :json
expect(response).to have_http_status(:forbidden)
sign_out
IpAddress.create!(ip_address: IPAddr.new('127.0.0.1').hton, banned_at: Time.current)
get '/static_mirror', headers: { 'REMOTE_ADDR' => '127.0.0.1' }
expect(response).to have_http_status(:forbidden)
expect(StaticMirrorJob).not_to have_received(:perform_later)
end
it 'requires the normal CSRF token for anonymous generation', csrf: true do
get '/users/csrf'
token = json.fetch('token')
post '/static_mirror/jobs', params: { locale: 'ja' }, as: :json
expect(response).to have_http_status(:forbidden)
post '/static_mirror/jobs', params: { locale: 'ja' }, as: :json,
headers: { 'X-CSRF-Token' => token }
expect(response).to have_http_status(:accepted)
end
end
+3 -3
ファイルの表示
@@ -55,7 +55,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do
end
it 'does not fall back to a revision absent from new history' do
versions.first.destroy!
TagDescriptionVersion.where(id: versions.first.id).delete_all
without_legacy_reads do
get "/wiki/#{ description.id }", params: { version: versions.first.id }
end
@@ -63,7 +63,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do
end
it 'does not fall back when the new current revision is missing' do
versions.last.destroy!
TagDescriptionVersion.where(id: versions.last.id).delete_all
without_legacy_reads { get "/wiki/#{ description.id }" }
expect(response).to have_http_status(:not_found)
end
@@ -190,7 +190,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do
describe 'index' do
[nil, 'description_contract'].each do |title|
it "returns new content once in #{ title ? 'filtered' : 'unfiltered' } index" do
get '/wiki', params: { title: }
get '/wiki', params: title.nil? ? { } : { title: title }
expect(response).to have_http_status(:ok)
rows = json.select { |row| row.fetch('id') == description.id }
expect(rows.size).to eq(1)
+319
ファイルの表示
@@ -0,0 +1,319 @@
require 'rails_helper'
require 'tmpdir'
require 'csv'
RSpec.describe 'Counts export contracts' do
include ActiveSupport::Testing::TimeHelpers
let(:payload) do
{ 'resource' => 'posts', 'definition' => 'history', 'mode' => 'details',
'locale' => 'ja', 'execution_at' => '2024-03-01T00:00:00Z',
'columns' => 'title', 'headers' => '["Title"]', 'format' => 'json' }
end
around do |example|
Dir.mktmpdir('counts-export-spec') do |path|
@counts_export_dir = Pathname.new(path)
HistoryAccess.set(user: nil) { example.run }
end
end
before do
allow(CountsExportDownload).to receive(:directory)
.and_return(@counts_export_dir)
end
context 'nico link IDs with the V1 export field name' do
let(:tag) { create(:tag, primary_name: 'linked export tag') }
let(:external) { create(:external_tag, name: 'external export tag') }
let(:payload) do
super().merge('resource' => 'nico_tags', 'columns' => 'linked_tags',
'headers' => '["linked_tags"]')
end
before do
NicoTagVersion.create!(external_tag: external, version_no: 1,
event_type: 'create', linked_tag_ids: '', created_at: Time.utc(2023))
NicoTagVersion.create!(external_tag: external, version_no: 2,
event_type: 'update', linked_tag_ids: tag.id.to_s, created_at: Time.utc(2024))
end
it 'exports recorded IDs with the legacy JSON key and metadata' do
result = JSON.parse(CountsTableExport.new(payload).body.to_a.join)
expect(result.fetch('data')).to eq([
{ 'linked_tags' => '' }, { 'linked_tags' => tag.id.to_s }])
expect(result.fetch('metadata').fetch('columns')).to eq(['linked_tags'])
keys = result.fetch('metadata').fetch('fields').map { _1.fetch('key') }
expect(keys).to include('linked_tags')
expect(keys).not_to include('linked_tag_ids')
end
it 'exports the selected legacy CSV header and ID values' do
body = CountsTableExport.new(payload.merge('format' => 'csv')).body.to_a.join
expect(CSV.parse(body.delete_prefix("\uFEFF")))
.to eq([['linked_tags'], [''], [tag.id.to_s]])
end
it 'keeps the legacy link column in default JSON exports' do
options = payload.except('columns', 'headers')
result = JSON.parse(CountsTableExport.new(options).body.to_a.join)
expect(result.fetch('metadata').fetch('columns')).to include('linked_tags')
result.fetch('data').each do |row|
expect(row).to have_key('linked_tags')
expect(row).not_to have_key('linked_tag_ids')
end
end
it 'rejects the physical column as a selected CSV or JSON field' do
['csv', 'json'].each do |format|
options = payload.merge('format' => format, 'columns' => 'linked_tag_ids')
expect { CountsTableExport.new(options) }.to raise_error(ArgumentError)
end
end
end
context 'serialization after the authorised query' do
let(:rows) { [{ title: "=formula\n\"quoted\"" }, { title: 'second' }] }
let(:page) do
{ count: 2, output_count: 2, visible_output_count: 2, page: 1, limit: 100,
versions: rows, execution_at: payload.fetch('execution_at') }
end
before do
query = CountsTableQuery.new(payload.slice(*CountsTableQuery::KEYS))
# SQL/authority are exercised in request specs; this unit boundary supplies their page.
allow(query).to receive(:each_page) do |before_page:, &block|
before_page.call
block.call(page)
end
allow(CountsTableQuery).to receive(:new).and_return(query)
end
it 'reports aggregation, actual output progress and regular-row metadata' do
progress = []
phases = []
body = CountsTableExport.new(payload).body(
progress: ->(done, total) { progress << [done, total] },
phase: ->(value) { phases << value }).to_a.join
result = JSON.parse(body)
expect(result['data']).to eq(rows.map(&:stringify_keys))
expect(result['metadata']).to include('logical_count_unit' => 'regular_result_rows',
'aggregate_records_in_logical_count' => false, 'output_count' => 2)
expect(phases).to eq(['aggregating', 'writing'])
expect(progress).to include([0, 2], [1, 2], [2, 2])
expect(progress.last).to eq([2, 2])
end
it 'preserves CSV multiline text and quotes while neutralising spreadsheet formulas' do
body = CountsTableExport.new(payload.merge('format' => 'csv')).body.to_a.join
expect(CSV.parse(body.delete_prefix("\uFEFF")))
.to eq([['Title'], ["'=formula\n\"quoted\""], ['second']])
end
it 'completes an isolated worker job with its real processed count and cancellable file' do
CountsExportJobs.pulse
token = CountsExportJobs.start(payload, 'owner', 'ja')
id = CountsExportJobs.authorised_id(token, 'owner')
state = CountsExportJobs.read(id).merge('status' => 'running', 'phase' => 'aggregating')
CountsExportJobs.write(state)
# Invoke the worker entrypoint directly; never fork or start an external worker in specs.
CountsExportJobs.perform(state)
ready = CountsExportJobs.status(token, 'owner')
expect(ready).to include('status' => 'ready', 'processed' => 2, 'total' => 2)
file = CountsExportDownload.find(ready.fetch('token'), 'owner')
expect(JSON.parse(File.read(file.fetch('path')))['data'].length).to eq(2)
CountsExportJobs.status(token, 'owner', cancel: true)
expect(CountsExportDownload.find(ready.fetch('token'), 'owner')).to be_nil
end
it 'marks a generation exception as failed without issuing a download token' do
CountsExportJobs.pulse
token = CountsExportJobs.start(payload, 'owner', 'ja')
id = CountsExportJobs.authorised_id(token, 'owner')
state = CountsExportJobs.read(id).merge('status' => 'running')
CountsExportJobs.write(state)
allow(CountsExportDownload).to receive(:prepare).and_raise(CountsExportDownload::TooLarge)
CountsExportJobs.perform(state)
expect(CountsExportJobs.status(token, 'owner')).to include('status' => 'failed')
expect(CountsExportJobs.status(token, 'owner')).not_to have_key('token')
end
end
it 'retains multilingual names and all inner rows in JSON with one selected display column' do
names = [{ language_code: 'ja', name: 'alpha', script_code: 'Jpan', aliases: [] },
{ language_code: 'ain', name: 'recorded', script_code: 'Kana',
aliases: [{ name: 'alias', script_code: 'Latn' }] }]
rows = [{ id: 1, name_ja: 'alpha', names: }, { id: 2, name_ja: 'beta', names: }]
projection = CountsExportProjection.new(payload.merge('columns' => 'tags',
'headers' => '["Tags"]', 'children' => 'tags.name_ja'),
resource: 'posts', summary: false, tags: false)
result = projection.project({ tags: rows })
expect(result['tags'].length).to eq(2)
expect(result['tags'].map { _1['name_ja'] }).to eq(['alpha', 'beta'])
expect(result['tags'].map { _1['names'] }).to eq([names, names])
end
it 'requires a live worker, enforces queue capacity and binds jobs to their owner' do
expect { CountsExportJobs.start(payload, 'owner', 'ja') }
.to raise_error(CountsExportJobs::WorkerUnavailable)
CountsExportJobs.pulse
token = CountsExportJobs.start(payload, 'owner', 'ja')
expect(CountsExportJobs.status(token, 'other')).to be_nil
expect(CountsExportJobs.status(token, 'owner')).to include(
'status' => 'queued', 'phase' => 'queued', 'processed' => 0, 'total' => nil)
(CountsExportDownload::MAX_CONCURRENT_EXPORTS - 1).times do
CountsExportJobs.start(payload, 'owner', 'ja')
end
expect { CountsExportJobs.start(payload, 'owner', 'ja') }
.to raise_error(CountsExportDownload::Busy)
expect(CountsExportJobs.status(token, 'owner', cancel: true)['status']).to eq('cancelled')
end
it 'fails stale queued/running jobs and preserves actual progress without a fake total' do
CountsExportJobs.pulse
token = CountsExportJobs.start(payload, 'owner', 'ja')
id = CountsExportJobs.authorised_id(token, 'owner')
state = CountsExportJobs.read(id)
state.merge!('status' => 'running', 'phase' => 'aggregating')
CountsExportJobs.write(state)
CountsExportJobs.checkpoint(id, 5, nil, phase: 'writing')
expect(CountsExportJobs.status(token, 'owner')).to include(
'phase' => 'writing', 'processed' => 5, 'total' => nil)
state = CountsExportJobs.read(id)
state['heartbeat_at'] = (Time.current - CountsExportJobs::HEARTBEAT_SECONDS - 1).to_f
CountsExportJobs.write(state)
expect(CountsExportJobs.status(token, 'owner')).to include(
'status' => 'failed', 'error' => 'worker_stalled', 'processed' => 5)
expect { CountsExportJobs.checkpoint(id, 6) }.to raise_error(CountsExportJobs::Cancelled)
end
it 'revokes an already generated download on a new discard and rejects wrong owners' do
exporter = instance_double(CountsExporter, resource: 'posts', format: 'json', body: ['{}'])
token = CountsExportDownload.prepare(exporter, 'owner')
expect(CountsExportDownload.find(token, 'owner')).not_to be_nil
expect(CountsExportDownload.find(token, 'other')).to be_nil
post = create(:post)
PostVersion.create!(post:, version_no: 1, event_type: 'discard', url: post.url,
title: 'private', tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024))
expect(CountsExportDownload.find(token, 'owner')).to be_nil
end
it 'rejects oversized output and removes its partial file' do
stub_const('CountsExportDownload::MAX_FILE_BYTES', 4)
exporter = instance_double(CountsExporter, resource: 'posts', format: 'json', body: ['12345'])
expect { CountsExportDownload.prepare(exporter, 'owner') }
.to raise_error(CountsExportDownload::TooLarge)
expect(Dir.glob(CountsExportDownload.directory.join('counts-*'))).to be_empty
end
it 'expires queued work after worker loss and rejects expired download signatures' do
CountsExportJobs.pulse
token = CountsExportJobs.start(payload, 'owner', 'ja')
travel_to(Time.current + CountsExportJobs::HEARTBEAT_SECONDS + 1) do
expect(CountsExportJobs.status(token, 'owner')).to include(
'status' => 'failed', 'error' => 'worker_unavailable')
end
exporter = instance_double(CountsExporter, resource: 'posts', format: 'json', body: ['{}'])
download = CountsExportDownload.prepare(exporter, 'owner')
travel_to(Time.current + CountsExportDownload::EXPIRY + 1) do
expect(CountsExportDownload.find(download, 'owner')).to be_nil
end
end
end
RSpec.describe 'Counts download acknowledgement' do
around do |example|
Dir.mktmpdir('counts-ack-spec') do |path|
@counts_ack_dir = Pathname.new(path)
HistoryAccess.set(user: nil) { example.run }
end
end
before do
allow(CountsExportDownload).to receive(:directory).and_return(@counts_ack_dir)
end
it 'binds an acknowledgement to the owner, job, file token and fresh attempt' do
CountsExportJobs.pulse
payload = { 'resource' => 'posts', 'definition' => 'history', 'mode' => 'details',
'format' => 'json', 'locale' => 'ja' }
token = CountsExportJobs.start(payload, 'owner', 'ja')
id = CountsExportJobs.authorised_id(token, 'owner')
state = CountsExportJobs.read(id).merge('status' => 'ready', 'token' => 'file-token')
CountsExportJobs.write(state)
receipt = CountsExportJobs.download_attempt(token, 'owner')
expect(receipt['download_status']).to eq('pending')
arguments = { file_token: 'file-token', attempt: receipt.fetch('download_attempt') }
expect(CountsExportJobs.download_attempt(token, 'other', **arguments)).to be_nil
expect(CountsExportJobs.download_attempt(token, 'owner', **arguments.merge(file_token: 'wrong'))).to be_nil
CountsExportJobs.download_attempt(token, 'owner', **arguments, status: 'streaming')
CountsExportJobs.download_attempt(token, 'owner', **arguments, status: 'started')
expect(CountsExportJobs.status(token, 'owner')['download_status']).to eq('started')
expect(CountsExportJobs.download_attempt(token, 'owner')['download_attempt'])
.to eq(receipt['download_attempt'])
CountsExportJobs.download_attempt(token, 'owner', **arguments, status: 'failed')
fresh = CountsExportJobs.download_attempt(token, 'owner')
expect(fresh['download_attempt']).not_to eq(receipt['download_attempt'])
expect(CountsExportJobs.download_attempt(token, 'owner', **arguments)).to be_nil
CountsExportJobs.status(token, 'owner', cancel: true)
expect(CountsExportJobs.download_attempt(token, 'owner')).to be_nil
end
end
RSpec.describe CountsExportDownload::ResponseBody do
it 'survives Rails buffer enumeration without closing before the first byte' do
Dir.mktmpdir('counts-body-spec') do |directory|
path = File.join(directory, 'file')
payload = 'data' * 20_000
File.binwrite(path, payload)
states = []
source = ActionDispatch::Response::FileBody.new(path)
body = described_class.new(source) { |state| states << state; true }
response = ActionDispatch::Response.new
response.body = body
_, _, rack_body = response.to_a
expect(states).to eq([])
chunks = []
rack_body.each { |chunk| chunks << chunk }
expect(chunks.join).to eq(payload)
expect(states.count('started')).to eq(1)
end
end
it 'acknowledges only after the first Rack yield and closes its bounded file body' do
Dir.mktmpdir('counts-body-spec') do |directory|
path = File.join(directory, 'file')
File.binwrite(path, 'x' * (64 * 1024 + 3))
states = []
source = ActionDispatch::Response::FileBody.new(path)
body = described_class.new(source) { |state| states << state; true }
chunks = []
body.each do |chunk|
expect(states).not_to include('started') if chunks.empty?
chunks << chunk
end
expect(chunks.join).to eq('x' * (64 * 1024 + 3))
expect(chunks.map(&:bytesize).max).to be <= 64 * 1024
expect(states.count('started')).to eq(1)
expect(body).not_to respond_to(:to_path)
expect { body.close }.not_to raise_error
end
end
it 'records a failed stream without converting an exception into a second HTTP response' do
Dir.mktmpdir('counts-body-spec') do |directory|
path = File.join(directory, 'file')
File.binwrite(path, 'data')
states = []
source = ActionDispatch::Response::FileBody.new(path)
body = described_class.new(source) { |state| states << state; true }
expect { body.each { raise IOError, 'connection closed' } }.to raise_error(IOError)
expect(states).to include('failed')
expect(states).not_to include('started')
expect { body.close }.not_to raise_error
end
end
end
+367
ファイルの表示
@@ -0,0 +1,367 @@
require 'rails_helper'
require 'csv'
RSpec.describe 'Counts masked pivot contracts' do
let(:admin) { create(:user, :admin) }
let(:member) { create(:user, :member) }
let(:guest) { create(:user, :guest) }
let(:params) do
{ 'resource' => 'posts', 'definition' => 'history', 'mode' => 'summary',
'locale' => 'ja', 'execution_at' => '2025-01-01T00:00:00Z',
'timezone' => 'UTC', 'rows' => 'tags', 'cols' => 'event_type', 'sort' => '' }
end
let!(:tags) do
['A', 'B', 'C'].map do |name|
tag = create(:tag, primary_name: name)
TagVersion.create!(tag:, version_no: 1, event_type: 'create', name:,
category: 'general', aliases: '', parent_tag_ids: '',
names: { ja: { name:, script_code: 'Jpan', auto_generated: false, aliases: [] } },
created_at: Time.utc(2023))
tag
end
end
before do
[[0, 1], [0], [1, 2]].each_with_index do |indexes, index|
post = create(:post)
snapshot = indexes.map { { tag_id: tags[_1].id, version_no: 1, sections: [] } }
PostVersion.create!(post:, version_no: 1, event_type: 'create',
title: "hidden-title-#{ index }", url: post.url, tags: '', tags_json: snapshot,
parent_post_ids: '', created_by_user: member, created_at: Time.utc(2024, 1))
next if index == 2
PostVersion.create!(post:, version_no: 2, event_type: 'discard',
title: "private-discard-#{ index }", url: post.url, tags: '', tags_json: [],
parent_post_ids: '', created_by_user: member, created_at: Time.utc(2024, 2))
end
end
def result(user, extra = {})
HistoryAccess.set(user:) { CountsTableQuery.new(params.merge(extra)).call }
end
def grand(result)
result.fetch(:pivot_totals).find { _1[:row].empty? && _1[:column].empty? }.fetch(:count)
end
it 'preserves monthly name subtotals and grand totals across pages' do
settings = { 'resource' => 'tags', 'rows' => 'created_at:month name' }
complete = result(admin, settings)
expect(complete[:group_count]).to eq(3)
expect(grand(complete)).to eq(3)
(1..3).each do |page|
paged = result(admin, settings.merge('limit' => '1', 'page' => page.to_s))
expect(grand(paged)).to eq(3)
subtotal = paged[:pivot_totals].find do |cell|
cell[:row] == ['2023-01'] && cell[:column].empty?
end
headers = paged[:row_headers].select { _1[:kind] == 'subtotal' }
if page == 3
expect(headers.map { _1[:values] }).to eq([['2023-01']])
expect(subtotal).not_to be_nil
expect(subtotal[:count]).to eq(3)
else
expect(headers).to eq([])
expect(subtotal).to be_nil
end
expect(paged[:cells].sum { _1[:count] }).to eq(1)
expect(paged[:group_count]).to eq(3)
end
end
it 'counts one opaque collection per private version across pages' do
[nil, guest, member, admin].each do |user|
page = result(user)
expected = user == admin ? 5 : 6
expect(grand(page)).to eq(expected)
expect(page[:cells].sum { _1[:count] }).to eq(expected)
expect(page).not_to have_key(:excluded_count)
paged = result(user, 'limit' => '1')
expect(grand(paged)).to eq(expected)
expect(result(user, 'limit' => '1', 'page' => '2')[:group_count])
.to eq(page[:group_count])
end
public_page = result(nil)
hidden = public_page[:cells].select { _1[:values].first == CountsAxes::MASKED_VALUE }
expect(hidden.sum { _1[:count] }).to eq(4)
expect(public_page[:axis_labels]['a0']).not_to have_key("internal:#{ tags.first.id }")
expect(public_page.to_json).not_to include('hidden-title', 'private-discard')
end
it 'retains distinct membership when tags are not a row axis' do
[nil, admin].each do |user|
expect(grand(result(user, 'rows' => 'event_type', 'cols' => 'tags')))
.to eq(user == admin ? 3 : 5)
expect(result(user, 'rows' => '', 'cols' => '')[:cells].sole[:count]).to eq(5)
details = result(user, 'mode' => 'details', 'rows' => '', 'cols' => '')
expect(details[:count]).to eq(user == admin ? 5 : 1)
end
end
it 'uses changed-record membership for totals and hides changes to private predecessors' do
post = PostVersion.order(:id).last.post
PostVersion.create!(post:, version_no: 2, event_type: 'update', title: 'changed-public-title',
url: post.url, tags: '',
tags_json: [{ tag_id: tags.last.id, version_no: 1, sections: [] }],
parent_post_ids: '', created_at: Time.utc(2024, 3))
restored = PostVersion.order(:id).first.post
PostVersion.create!(post: restored, version_no: 3, event_type: 'restore',
title: 'restored-public-title', url: restored.url, tags: '',
tags_json: [{ tag_id: tags.first.id, version_no: 1, sections: [] }], parent_post_ids: '',
created_at: Time.utc(2024, 4))
[nil, admin].each do |user|
page = result(user, 'rows' => 'tags changed_field', 'cols' => 'event_type')
expect(grand(page)).to eq(page[:cells].sum { _1[:count] })
expect(page[:cells]).not_to be_empty
if user.nil?
expect(page[:cells].any? { _1[:values][1] == CountsAxes::MASKED_VALUE }).to be(true)
end
expect(page.to_json).not_to include('hidden-title', 'private-discard')
end
end
it 'counts distinct changed records when tags are not additive row tuples' do
[nil, admin].each do |user|
page = result(user, 'rows' => 'changed_field', 'cols' => 'event_type')
expect(grand(page)).to eq(user == admin ? 2 : 4)
if user.nil?
expect(page[:cells].map { _1[:values].first }.uniq).to eq([CountsAxes::MASKED_VALUE])
end
# Private empty collections contribute the same opaque tuple as populated ones.
combined = result(user, 'rows' => 'changed_field', 'cols' => 'tags')
expect(grand(combined)).to eq(user == admin ? 0 : 4)
end
end
it 'keeps NULL as a public data value rather than treating it as a rollup' do
post = create(:post)
PostVersion.create!(post:, version_no: 1, event_type: 'create', title: 'public bot',
url: post.url, tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024, 5))
page = result(nil, 'rows' => 'user', 'cols' => 'event_type')
expect(page[:cells].select { _1[:values].first.nil? }.sum { _1[:count] }).to eq(1)
expect(grand(page)).to eq(6)
expect(page[:row_headers]).to include(hash_including(values: [nil], kind: 'data'))
end
it 'sorts public version numbers numerically including formerly hidden records' do
post = PostVersion.order(:id).last.post
[2, 10].each do |number|
PostVersion.create!(post:, version_no: number, event_type: 'update',
title: 'public numeric sort', url: post.url, tags: '', tags_json: [],
parent_post_ids: '', created_at: Time.utc(2024, 5))
end
page = result(nil, 'rows' => 'version_no', 'cols' => '')
values = page[:row_headers].select { _1[:kind] == 'data' }.map { _1[:values].first }
expect(values.map(&:to_i)).to eq([1, 2, 10])
end
it 'keeps public identity predicates and excludes private attributes' do
hidden_id = PostVersion.first.post_id.to_s
base = { 'rows' => '', 'cols' => '' }
condition = lambda do |field, operator, value|
{ 'conditions' => '', 'conditions.0.field' => field,
'conditions.0.operator' => operator, 'conditions.0.value' => value }
end
expect(result(nil, base.merge('resource_id' => hidden_id))[:cells].sole[:count]).to eq(2)
expect(result(nil, base.merge(condition.call('version_no', 'eq', '2')))[:cells]
.sole[:count]).to eq(2)
expect(result(nil, base.merge(condition.call('user', 'eq', member.id.to_s)))[:cells]
.sole[:count]).to eq(5)
expect(result(nil, base.merge(condition.call('title', 'eq', 'hidden-title-0')))[:cells]
.sole[:count]).to eq(0)
expect(result(nil, base.merge(condition.call('title', 'ne', 'hidden-title-0')))[:cells]
.sole[:count]).to eq(1)
expect(result(nil, base.merge(condition.call('title', 'eq', '********')))[:cells]
.sole[:count]).to eq(0)
expect(result(nil, base.merge(condition.call('video_ms', 'eq', '********')))[:cells]
.sole[:count]).to eq(0)
expect(result(nil, base.merge(condition.call('title', 'like', '%title%')))[:cells]
.sole[:count]).to eq(1)
expect(result(nil, base.merge(condition.call('title', 'like', '%')))[:cells]
.sole[:count]).to eq(1)
expect(result(nil, base.merge('tags' => 'C'))[:cells].sole[:count]).to eq(1)
expect(result(nil, base.merge('tags' => 'not:C'))[:cells].sole[:count]).to eq(0)
end
it 'retains public OR branches while rejecting private predicates on hidden rows' do
HistoryAccess.set(user: nil) do
population = CountsPopulation.new(params.merge('rows' => '', 'cols' => ''),
authorisation_flg: false)
condition = CountsCondition.new(population)
public_scope = condition.apply(population.scope,
{ 'field' => 'event_type', 'operator' => 'eq', 'value' => 'discard' })
private_scope = condition.apply(population.scope,
{ 'field' => 'title', 'operator' => 'ne', 'value' => 'unrelated' })
masked_scope = condition.apply(population.scope,
{ 'field' => 'title', 'operator' => 'eq', 'value' => '********' })
expect(public_scope.except(:select).count).to eq(2)
expect(public_scope.or(private_scope).except(:select).count).to eq(3)
expect(private_scope.or(public_scope).except(:select).count).to eq(3)
expect(public_scope.or(masked_scope).except(:select).count).to eq(2)
expect(public_scope.merge(private_scope).except(:select).count).to eq(0)
end
end
it 'excludes private collections from positive, negative, AND and OR tag searches' do
settings = { 'rows' => '', 'cols' => '' }
{ 'C' => 1, 'not:C' => 0, 'not:unknown' => 1, '********' => 0,
'not:********' => 1, 'not:C C' => 0, 'not:C type:or C' => 1 }.each do |value, count|
expect(result(nil, settings.merge('tags' => value))[:cells].sole[:count]).to eq(count)
end
end
it 'ignores empty numeric and interval bounds without restricting private history' do
settings = { 'rows' => '', 'cols' => '', 'conditions' => '' }
{ 'video_ms' => ['eq', 'ne', 'gt', 'ge', 'lt', 'le', 'range'],
'tags.sections' => ['range'] }.each do |field, operators|
operators.each do |operator|
condition = settings.merge('conditions.0.field' => field,
'conditions.0.operator' => operator)
empty = condition.merge('conditions.0.value' => '')
expect(result(nil, condition)[:cells].sole[:count]).to eq(5)
expect(result(nil, empty)[:cells].sole[:count]).to eq(5)
expect(result(nil, empty.merge('conditions.0.blank' => '0'))[:cells]
.sole[:count]).to eq(5)
if operator == 'range'
empty['conditions.0.end'] = ''
expect(result(nil, empty)[:cells].sole[:count]).to eq(5)
['conditions.0.value', 'conditions.0.end'].each do |key|
active = empty.merge(key => '100')
expect(result(nil, active)[:cells].sole[:count]).to eq(0)
end
end
end
end
end
it 'keeps typed marker sets active after incompatible marker values are removed' do
settings = { 'rows' => '', 'cols' => '', 'conditions' => '',
'conditions.0.field' => 'video_ms', 'conditions.0.values' => '1',
'condition_values' => '', 'condition_values.0.condition' => '0',
'condition_values.0.value_type' => 'string',
'condition_values.0.value' => '********' }
['in', 'not_in'].each do |operator|
expected = operator == 'in' ? 0 : 1
condition = settings.merge('conditions.0.operator' => operator)
expect(result(nil, condition)[:cells].sole[:count]).to eq(expected)
end
end
it 'restricts active numeric comparisons and preserves empty selection semantics' do
settings = { 'rows' => '', 'cols' => '', 'conditions' => '',
'conditions.0.field' => 'video_ms' }
['eq', 'ne', 'gt', 'ge', 'lt', 'le', 'range'].each do |operator|
condition = settings.merge('conditions.0.operator' => operator,
'conditions.0.value' => '0')
condition['conditions.0.end'] = '100' if operator == 'range'
# The one public row is NULL; private NULLs and real values never match.
expected = operator == 'ne' ? 1 : 0
expect(result(nil, condition)[:cells].sole[:count]).to eq(expected)
end
['in', 'not_in'].each do |operator|
condition = settings.merge('conditions.0.operator' => operator,
'conditions.0.values' => '1', 'condition_values' => '')
expected = operator == 'not_in' ? 5 : 0
expect(result(nil, condition)[:cells].sole[:count]).to eq(expected)
end
blank = settings.merge('conditions.0.operator' => 'eq', 'conditions.0.blank' => '1')
expect(result(nil, blank)[:cells].sole[:count]).to eq(1)
end
it 'preserves empty tag sets and restricts active blank collection predicates' do
settings = { 'rows' => '', 'cols' => '', 'conditions' => '',
'conditions.0.field' => 'tags' }
['any', 'all', 'none'].each do |operator|
condition = settings.merge('conditions.0.operator' => operator,
'conditions.0.values' => '1', 'condition_values' => '')
expected = operator == 'any' ? 0 : 5
expect(result(nil, condition)[:cells].sole[:count]).to eq(expected)
end
blank = settings.merge('conditions.0.operator' => 'eq', 'conditions.0.blank' => '1')
expect(result(nil, blank)[:cells].sole[:count]).to eq(0)
end
it 'rejects server-only coordinates in column order' do
probes = [
{ 'cols' => 'tags', 'column_order' => '', 'column_order.0.value_type' => 'string',
'column_order.0.value' => CountsAxes::MASKED_VALUE }]
HistoryAccess.set(user: nil) do
probes.each do |probe|
expect { CountsTableQuery.new(params.merge(probe)).call }
.to raise_error(CountsAxes::PrivateCondition)
end
end
end
it 'keeps private collection and change counts independent of stored elements' do
settings = [
{ 'rows' => 'tags', 'cols' => 'changed_field' },
{ 'rows' => 'changed_field tags', 'cols' => '' }]
original = settings.map { result(nil, _1) }
snapshot = tags.map do |tag|
{ tag_id: tag.id, version_no: 1, sections: [] }
end
hidden_posts = PostVersion.where(event_type: 'discard').pluck(:post_id)
PostVersion.where(post_id: hidden_posts).update_all(
title: 'same hidden value', tags_json: snapshot)
expect(settings.map { result(nil, _1) }).to eq(original)
end
it 'excludes private tag and change coordinates from drill conditions' do
['tags', 'changed_field'].each do |field|
settings = { 'rows' => field, 'cols' => '' }
page = result(nil, settings)
cell = page[:cells].find { _1[:values] == [CountsAxes::MASKED_VALUE] }
expect(cell[:count]).to eq(4)
[CountsAxes::MASKED_VALUE, CountsAxes::MASKED_TEXT].each do |value|
drill = { 'drill' => '', 'drill.0.field' => field,
'drill.0.value_type' => 'string', 'drill.0.value' => value }
expect(result(nil, settings.merge(drill))[:cells]).to be_empty
details = result(nil, settings.merge(drill).merge('mode' => 'details'))
expect(details[:versions]).to be_empty
condition = { 'conditions' => '', 'conditions.0.field' => field,
'conditions.0.operator' => 'axis_eq',
'conditions.0.value_type' => 'string', 'conditions.0.value' => value }
expect(result(nil, settings.merge(condition))[:cells]).to be_empty
end
end
end
it 'rejects real private tag and change coordinates for non-admin users' do
[nil, guest, member].each do |user|
{ 'tags' => "internal:#{ tags.first.id }", 'changed_field' => 'title' }.each do
|field, value|
settings = { 'rows' => '', 'cols' => '', 'drill' => '',
'drill.0.field' => field, 'drill.0.value_type' => 'string',
'drill.0.value' => value }
expect(result(user, settings)[:cells].sole[:count]).to eq(0)
expect(result(admin, settings)[:cells].sole[:count]).to be > 0
end
end
end
it 'keeps public column sorting usable while guessed private coordinates match nothing' do
first = result(nil, 'rows' => 'event_type', 'cols' => 'tags')
probe = result(nil, 'rows' => 'event_type', 'cols' => 'tags',
'column_order' => '', 'column_order.0.value_type' => 'string',
'column_order.0.value' => "internal:#{ tags.first.id }")
expect(probe[:column_values]).to eq(first[:column_values])
expect(probe[:cells]).to eq(first[:cells])
expect(probe[:column_values_editable]).to be(false)
expect(probe.to_json).not_to include("internal:#{ tags.first.id }")
end
it 'exports masked coordinates and safe CSV labels from the same pivot' do
HistoryAccess.set(user: nil) do
payload = params.merge('format' => 'json')
json = JSON.parse(CountsTableExport.new(payload).body.to_a.join)
expect(json['data'].sum { _1['kind'] == 'data' ? _1['count'] : 0 }).to eq(6)
expect(json.to_json).not_to include('hidden-title', 'private-discard')
csv = CountsTableExport.new(payload.merge('format' => 'csv')).body.to_a.join
expect(csv).to include('********')
expect(csv).not_to include(CountsAxes::MASKED_VALUE, 'hidden-title', 'private-discard')
end
end
end
+18 -8
ファイルの表示
@@ -8,21 +8,23 @@ RSpec.describe NicoTagVersionRecorder do
described_class.record!(external_tag:, event_type:, created_by_user: member)
end
it 'records the external association and platform-qualified name' do
it 'records the external association and derives the platform-qualified name' do
version = record(:create)
expect(version).to have_attributes(
external_tag:, version_no: 1, event_type: 'create',
name: 'nico:raw tag[]', linked_tags: '', created_by_user: member)
name: 'nico:raw tag[]', linked_tag_ids: '', created_by_user: member)
expect(external_tag.reload.nico_tag_versions).to contain_exactly(version)
end
it 'uses the latest history number when the record has no version_no column' do
first = record(:create)
external_tag.update!(name: 'changed')
tag = create(:tag)
NicoTagRelation.create!(nico_tag: external_tag, tag:)
second = record(:update)
expect(second).to have_attributes(version_no: 2, name: 'nico:changed')
expect(second).to have_attributes(version_no: 2, name: 'nico:raw tag[]',
linked_tag_ids: tag.id.to_s)
expect(first.reload.name).to eq('nico:raw tag[]')
expect(external_tag.reload.has_attribute?(:version_no)).to be(false)
end
@@ -40,7 +42,7 @@ RSpec.describe NicoTagVersionRecorder do
expect(version).to have_attributes(
external_tag: external, name: 'registered_external:foo', version_no: 1,
event_type: 'create', linked_tags: '', created_by_user: member)
event_type: 'create', linked_tag_ids: '', created_by_user: member)
end
it 'returns the latest version without appending an unchanged snapshot' do
@@ -64,13 +66,21 @@ RSpec.describe NicoTagVersionRecorder do
expect(external_tag.nico_tag_versions).to contain_exactly(first)
end
it 'records sorted linked internal names and later link removal' do
it 'rejects discard and restore events for external tags' do
[:discard, :restore].each do |event_type|
expect { record(event_type) }
.to raise_error(ArgumentError, "Invalid event_type: #{ event_type }")
end
end
it 'records sorted linked internal IDs and later link removal' do
tags = ['z_link', 'a_link'].map { |name| create(:tag, primary_name: name, category: :general) }
tags.each { |tag| NicoTagRelation.create!(nico_tag: external_tag, tag:) }
expect(record(:create).linked_tags).to eq('a_link z_link')
expect(record(:create).linked_tag_ids).to eq(tags.map(&:id).sort.join(' '))
external_tag.linked_tags = [tags.first]
expect(record(:update)).to have_attributes(version_no: 2, linked_tags: 'z_link')
expect(record(:update)).to have_attributes(version_no: 2, linked_tag_ids: tags.first.id.to_s)
end
end
+33
ファイルの表示
@@ -0,0 +1,33 @@
require 'rails_helper'
RSpec.describe QueryState do
it 'decodes typed axis values and keeps absence distinct from an empty set' do
{ 'boolean' => ['false', false], 'number' => ['7', 7],
'string' => ['a b & c', 'a b & c'], 'null' => ['', nil] }.each do |type, (raw, expected)|
params = { 'conditions' => '', 'conditions.0.field' => 'user',
'conditions.0.operator' => 'axis_eq',
'conditions.0.value_type' => type, 'conditions.0.value' => raw }
expect(described_class.records(params, 'conditions').sole['value']).to eq(expected)
end
expect(described_class.records({}, 'conditions')).to eq([])
expect(described_class.records({ 'conditions' => '' }, 'conditions')).to eq([])
expect(described_class.records({ 'conditions' => '', 'conditions.0.values' => '1' },
'conditions')).to eq([{ 'values' => true }])
end
it 'rejects gaps, unknown keys, JSON, wrong types and oversized lists' do
[
{ 'conditions' => '[]' }, { 'conditions' => '', 'conditions.1.field' => 'user' },
{ 'conditions' => '', 'conditions.0.sql' => 'TRUE' },
{ 'conditions' => '', 'conditions.0.blank' => 'true' },
{ 'conditions' => '', 'conditions.0.value' => ['x'] },
{ 'drill' => '', 'drill.0.field' => 'user', 'drill.0.value_type' => 'object' },
{ 'conditions' => 'x' * 131_073 }].each do |params|
expect { described_class.validate!(params, ['conditions', 'drill']) }
.to raise_error(ArgumentError)
end
['user user', 'user title', ' user', (1..201).to_a.join(' ')].each do |value|
expect { described_class.list(value) }.to raise_error(ArgumentError)
end
end
end
+18 -2
ファイルの表示
@@ -8,6 +8,10 @@ RSpec.describe 'Shared resource IDs across MySQL connections', type: :model do
expect(Rails.env.test?).to be(true)
expect(ApplicationRecord.connection.adapter_name).to match(/mysql/i)
expect(ApplicationRecord.connection_pool.size).to be >= 3
@main_connection = ApplicationRecord.connection
# A leaked fixture pin makes worker threads share the main session. Fail before
# starting a worker rather than diagnosing IS_USED_LOCK(NULL) after a timeout.
expect(@main_connection.open_transactions).to eq(0)
@prefix = "shared_ids_#{ SecureRandom.hex(8) }"
@tag = create(:tag, primary_name: "#{ @prefix }_owner")
@group = TagGroup.create!(name: "#{ @prefix }_owner", version_no: 1)
@@ -42,13 +46,20 @@ RSpec.describe 'Shared resource IDs across MySQL connections', type: :model do
end
def await_queue queue
Timeout.timeout(15) { queue.pop }
value = Timeout.timeout(15) { queue.pop }
raise value if value.is_a?(Exception)
value
end
def worker &block
Thread.new do
Thread.current.report_on_exception = false
ApplicationRecord.connection_pool.with_connection(&block)
ApplicationRecord.connection_pool.with_connection do |connection|
raise 'Shared ID worker reused the main connection' if connection.equal?(@main_connection)
block.call(connection)
end
end.tap { @threads << _1 }
end
@@ -86,11 +97,16 @@ RSpec.describe 'Shared resource IDs across MySQL connections', type: :model do
ActiveSupport::Notifications.subscribed(subscriber, 'sql.active_record') do
id = create_resource(kind).id
end
raise 'Shared ID creation did not acquire a named lock' unless lock_name
ready << [id, connection.select_value('SELECT CONNECTION_ID()').to_i, lock_name]
await_queue(@release)
raise ActiveRecord::Rollback if rollback
end
id
rescue StandardError => error
ready << error
raise
end
[thread, await_queue(ready)]
end
+249
ファイルの表示
@@ -0,0 +1,249 @@
require 'rails_helper'
RSpec.describe StaticMirror do
include_context 'Ainu locale'
include ActiveSupport::Testing::TimeHelpers
let(:tag) { create(:tag, primary_name: 'mirror-public-tag') }
let(:post) { create(:post, title: '<img src=x onerror=alert(1)>') }
let(:mirror) { described_class.new(locale: Locale.nipponese, origin: 'https://source.test') }
def snapshot_tag(record, event: 'create', number: 1)
TagVersion.create!(tag: record, version_no: number, event_type: event,
name: record.name('ja'), category: record.category, aliases: '', parent_tag_ids: '',
names: { ja: { name: record.name('ja'), script_code: 'Jpan',
auto_generated: false, aliases: [] } }, created_at: Time.current)
end
def snapshot_post(record, tags: [])
PostVersion.create!(post: record, version_no: 1, event_type: 'create',
title: record.title, url: record.url, tags: '', tags_json: tags,
parent_post_ids: '', created_at: Time.current)
end
# Read STORE entries independently of the production writer and verify CRCs.
def archive_files(bytes)
offset = 0
files = { }
while bytes.byteslice(offset, 4).unpack1('V') == 0x04034b50
header = bytes.byteslice(offset, 30).unpack('VvvvvvVVVvv')
size, name_size, extra_size = header[7], header[9], header[10]
path = bytes.byteslice(offset + 30, name_size).force_encoding('UTF-8')
data = bytes.byteslice(offset + 30 + name_size + extra_size, size)
expect(Zlib.crc32(data)).to eq(header[6])
files[path.delete_prefix('archive/')] = data.force_encoding('UTF-8')
offset += 30 + name_size + extra_size + size
end
expect(bytes.byteslice(-22, 4).unpack1('V')).to eq(0x06054b50)
files
end
before do
snapshot_tag(tag)
PostTag.create!(post:, tag:)
snapshot_post(post, tags: [{ tag_id: tag.id, version_no: 1, sections: [] }])
end
it 'generates a navigable ZIP, escaped HTML, search data and verified hashes' do
progress = []
files = archive_files(mirror.export { |done, total| progress << [done, total] })
expect(progress.last).to eq([2, 2])
expect(files.keys).to include('index.html', 'robots.txt', 'assets/archive.css',
'assets/archive.js', 'archive-config.js', 'search-index.json', 'manifest.json',
'ja/posts/index.html', "ja/posts/#{ post.id }/index.html",
'ja/tags/index.html', "ja/tags/#{ tag.id }/index.html")
files.select { |path, _| path.end_with?('.html') }.each do |path, html|
document = Nokogiri::HTML(html)
expect(document.at_css('meta[name="robots"]')['content']).to include('noindex')
expect(document.text).to include('静的保存版', mirror.data_at)
expect(document.css('img')).to be_empty
expect(document.at_css('[data-original]')['href']).to start_with('https://source.test/ja')
document.css('a[href], link[href], script[src]').each do |node|
target = node['href'] || node['src']
next if target.start_with?('https://')
expect(target).not_to start_with('/')
resolved = Pathname.new(path).dirname.join(target).cleanpath.to_s
expect(files).to have_key(resolved)
end
end
detail = Nokogiri::HTML(files.fetch("ja/posts/#{ post.id }/index.html"))
expect(detail.at_css('h1').text).to eq(post.title)
expect(detail.css('main a').map { _1['href'] })
.to include("../../tags/#{ tag.id }/index.html")
manifest = JSON.parse(files.fetch('manifest.json'))
expect(manifest).to include('format_version' => 1, 'locale' => 'ja',
'data_at' => mirror.data_at, 'data_at_basis' => 'capture_started_at',
'resources' => { 'posts' => 1, 'tags' => 1 })
expect(Time.iso8601(manifest.fetch('generated_at'))).to be >= Time.iso8601(mirror.data_at)
expect(manifest.fetch('files').map { _1.fetch('path') })
.to match_array(files.keys - ['manifest.json'])
manifest.fetch('files').each do |entry|
digest = Digest::SHA256.hexdigest(files.fetch(entry.fetch('path')))
expect(entry.fetch('sha256')).to eq(digest)
end
expect(JSON.parse(files.fetch('search-index.json')).map { _1.fetch('path') })
.to include("ja/posts/#{ post.id }/index.html", "ja/tags/#{ tag.id }/index.html")
end
it 'keeps private and unversioned records out even for an admin caller' do
private_tag = create(:tag, primary_name: 'mirror-secret-tag')
snapshot_tag(private_tag)
snapshot_tag(private_tag, event: 'discard', number: 2)
private_post = create(:post, title: 'mirror-secret-post')
snapshot_post(private_post)
PostVersion.create!(post: private_post, version_no: 2, event_type: 'discard',
url: private_post.url, tags: '', tags_json: [], parent_post_ids: '',
created_at: Time.current)
legacy = create(:post, title: 'mirror-unversioned-secret')
referencing = create(:post, title: 'mirror-secret-reference')
snapshot_post(referencing,
tags: [{ tag_id: private_tag.id, version_no: 1, sections: [] }])
files = HistoryAccess.set(user: create(:user, :admin)) { archive_files(mirror.export) }
expect(files.values.join).not_to include('mirror-secret', legacy.title, private_post.url)
expect(files.keys).not_to include("ja/tags/#{ private_tag.id }/index.html",
"ja/posts/#{ private_post.id }/index.html", "ja/posts/#{ referencing.id }/index.html")
expect(mirror.authority.fetch('publication_policy'))
.to eq('static-mirror-public-projection-1')
expect(mirror.authority.keys).not_to include('admin', 'actor_id', 'artifact_policy')
end
it 'applies later discard boundaries after a restoration' do
snapshot_tag(tag, event: 'discard', number: 2)
snapshot_tag(tag, event: 'restore', number: 3)
tag.update_columns(version_no: 3)
expect(archive_files(mirror.export)).to have_key("ja/tags/#{ tag.id }/index.html")
snapshot_tag(tag, event: 'discard', number: 4)
second = described_class.new(locale: Locale.nipponese, origin: 'https://source.test')
files = archive_files(second.export)
expect(files.keys).not_to include("ja/tags/#{ tag.id }/index.html")
expect(files.values.join).not_to include('mirror-public-tag')
end
it 'does not include credentials or executable URL schemes' do
post.update_columns(url: 'https://secret-user:secret-password@source.test/path')
content = archive_files(mirror.export).values.join
expect(content).not_to include('secret-password', 'secret-user')
post.update_columns(url: 'javascript:alert(1)')
expect(archive_files(mirror.export).values.join).not_to include('javascript:')
end
it 'fails if publication changes during generation' do
expect do
mirror.export do |done, _|
snapshot_tag(tag, event: 'discard', number: 2) if done.zero?
end
end.to raise_error(HistoryAccess::Forbidden)
end
it 'owns the static publication contract independently of Counts' do
allow(HistoryAccess).to receive(:authority).and_raise('Counts metadata is not applicable')
allow(HistoryAccess).to receive(:authorised_artifact?).and_return(false)
archive_files(mirror.export)
expect(described_class.authorised_artifact?(mirror.authority)).to be(true)
metadata = mirror.authority.except('publication_policy')
.merge('artifact_policy' => HistoryAccess::ARTIFACT_POLICY)
expect(described_class.authorised_artifact?(metadata)).to be(false)
stub_const('HistoryAccess::ARTIFACT_POLICY', 'another-counts-policy')
expect(described_class.authorised_artifact?(mirror.authority)).to be(true)
end
it 'rejects stale projections after edits without discard, including for admins' do
create(:tag_name, :alias, tag:, name: 'mirror-old-alias')
archive_files(mirror.export)
HistoryAccess.set(user: create(:user, :admin)) do
expect(described_class.authorised_artifact?(mirror.authority)).to be(true)
TagName.find_by!(tag:, name: 'mirror-old-alias').update!(name: 'mirror-new-alias')
expect(described_class.authorised_artifact?(mirror.authority)).to be(false)
end
end
it 'rejects a deprecated tag and its names without requiring a discard' do
archive_files(mirror.export)
tag.update!(deprecated_at: Time.current)
expect(described_class.authorised_artifact?(mirror.authority)).to be(false)
files = archive_files(mirror.export)
expect(files.keys).not_to include("ja/tags/#{ tag.id }/index.html")
expect(files.values.join).not_to include('mirror-public-tag')
end
it 'rejects stale post content and relations without a discard' do
archive_files(mirror.export)
post.update!(title: 'mirror-corrected-title')
expect(described_class.authorised_artifact?(mirror.authority)).to be(false)
archive_files(mirror.export)
PostTag.where(post:, tag:).delete_all
expect(described_class.authorised_artifact?(mirror.authority)).to be(false)
end
it 'rejects an artifact after its locale is deprecated' do
archive_files(mirror.export)
Locale.nipponese.update!(deprecated_at: Time.current)
expect(described_class.authorised_artifact?(mirror.authority)).to be(false)
end
it 'fails if a tag is deprecated during generation' do
expect do
mirror.export do |processed, _total|
tag.update!(deprecated_at: Time.current) if processed.zero?
end
end.to raise_error(HistoryAccess::Forbidden)
end
it 'records capture entry rather than claiming the exact DB read-view time' do
freeze_time do
capture_at = Time.current.utc.iso8601(6)
allow(CountsExporter).to receive(:with_read_snapshot).and_wrap_original do |original, &block|
travel 5.seconds
original.call(&block)
end
files = archive_files(mirror.export)
manifest = JSON.parse(files.fetch('manifest.json'))
expect(manifest).to include('data_at' => capture_at, 'data_at_basis' => 'capture_started_at')
expect(Time.iso8601(manifest.fetch('generated_at'))).to be > Time.iso8601(capture_at)
end
end
it 'bounds output and never reports hidden population counts' do
stub_const('StaticMirror::RESOURCE_LIMIT', 1)
second = create(:post, title: 'mirror-beyond-limit')
snapshot_post(second)
files = archive_files(mirror.export)
expect(files.values.join).not_to include(second.title, 'population_count', 'excluded_count')
stub_const('StaticMirror::MAX_BYTES', 16)
expect { mirror.export }.to raise_error(StaticMirror::TooLarge)
end
it 'keeps all registered language names and aliases without locale filtering' do
locale
create(:tag_name, :alias, tag:, name: 'mirror-alias', language_code: 'ja', script_code: 'Jpan')
files = archive_files(described_class.new(locale:, origin: 'https://source.test').export)
detail = files.fetch("ain-Kana/tags/#{ tag.id }/index.html")
expect(detail).to include('mirror-public-tag', 'mirror-alias', 'lang="ain"', 'lang="ja"')
index = JSON.parse(files.fetch('search-index.json'))
entry = index.find { _1.fetch('path') == "ain-Kana/posts/#{ post.id }/index.html" }
expect(entry.fetch('names')).to include('mirror-public-tag', 'mirror-alias')
end
it 'indexes only names of public linked tags within the output limit' do
other = create(:tag, primary_name: 'mirror-unlinked-tag')
snapshot_tag(other)
private_tag = create(:tag, primary_name: 'mirror-private-tag')
snapshot_tag(private_tag, event: 'discard')
PostTag.create!(post:, tag: private_tag)
files = archive_files(mirror.export)
index = JSON.parse(files.fetch('search-index.json'))
entry = index.find { _1.fetch('path') == "ja/posts/#{ post.id }/index.html" }
expect(entry.fetch('names')).to include('mirror-public-tag')
expect(entry.fetch('names')).not_to include('mirror-private-tag', 'mirror-unlinked-tag')
stub_const('StaticMirror::RESOURCE_LIMIT', 1)
expect(archive_files(mirror.export).fetch('search-index.json'))
.not_to include('mirror-unlinked-tag', 'mirror-private-tag')
end
it 'fails rather than truncating registered aliases at the name limit' do
create(:tag_name, :alias, tag:, name: 'mirror-extra-alias')
stub_const('StaticMirror::NAME_LIMIT', 1)
expect { mirror.export }.to raise_error(StaticMirror::TooLarge)
end
end
+3366
ファイルの表示
ファイル差分が大きすぎるため省略します 差分を読み込み
+70 -5
ファイルの表示
@@ -35,7 +35,9 @@ Web とワーカーは同一リリース、Rails secret、アプリケーショ
- POST `/counts/export/jobs`:要求検証・実行時点の固定・署名付きジョブトークンの発行。
- GET `/counts/export/jobs/:token`:状態、実際の処理済みレコード数、判明している総数。
- DELETE `/counts/export/jobs/:token`:取消を永続記録。
- GET `/counts/export/files/:token`:生成完了後の署名付きファイルトークンによる取得。
- POST `/counts/export/jobs/:token/download`:セッション所有者の開始確認要求を登録。
- GET `/counts/export/files/:token?job=...&attempt=...`:実 GET の認可・送出開始の確認。
- GET `/counts/export/files/:token`:既存クライアント向けの認可付きダウンロード。
ジョブ・ファイルのトークンは作成元のセッションに紐付く。
既存の BAN・CSRF 制御を使用する。フロントは約800ms間隔で状態を取得する。
@@ -80,16 +82,22 @@ processed は CSV では実際に生成したデータ行数、JSON のクロス
通常の Counts 応答では、使用していない全履歴の population_count は計算しない。
キーは null、population_count_computed は false とし、0に置き換えない。
現在条件の output_count は権限適用前、visible_output_count は権限適用後の
通常の集計行数(明細では明細行数)。小計・合計の物理行は含めない。
summary は非公開履歴を含む同じ母集団を集計する。非公開の軸はサーバーの CASE 式で
単一の予約値へまとめてから GROUP BY し、output_count / visible_output_count はその
安全化した通常行数を表す。excluded_count は返さない。details は従来どおり
output_count が権限適用前、visible_output_count が適用後の明細行数である。
小計・合計の物理行は行数に含めない。
CSV の物理行と JSON の pivot_cell には集計レコードを含むため、両者を混同しない。
除票境界・参照禁止集合は GROUP BY で識別キー単位にまとめる。
投稿の参照判定は、全除票履歴を JSON_CONTAINS / FIND_IN_SET で反復照合せず、
保存済みスナップショットから参照IDを JSON_TABLE で展開して照合する。
権限条件付きの母集団件数は同一 export インスタンス内で再利用する。
多段小計は、認可済み母集団の CTE を MySQL の無制限 LIMIT で実体化させ、
各枝が同じ認可判定をやり直すことを避ける。計数は COUNT(DISTINCT v.id) のまま。
多段小計は、全軸の JOIN へ実際に寄与するレコードを CTE に保持し、各枝で同じ
母集合を使う。changed_field には直前版がある実変更だけが寄与する。タグが行軸に
ない場合は DISTINCT 投稿版を数える。タグが行軸にある場合は投稿版×タグを数え、
全行軸の葉単位から小計・合計を求める。変更項目も行軸ならその寄与も保持する。
private のタグ・変更項目をまとめても、内部の寄与は捨てず、ラベルだけを保護する。
スキーマ上、以下の追加インデックスが候補となる。migration は作成・実行していない。
@@ -105,3 +113,60 @@ CSV の物理行と JSON の pivot_cell には集計レコードを含むため
追加は許可後の新規 add_index migration として行い、既存 migration は変更しない。
実DBでのインデックス容量・作成時ロック・実行計画・時間は未確認。
索引追加なしでの最大データ量に対する応答時間は保証しない。
## GET 応答開始の確認(静的実装・未実測)
進捗画面は ready 後に owner/job に紐づく download_attempt を登録し、同じ attempt を
native iframe の実 GET に渡す。サーバーは署名・所有者・権限 revision・期限・ファイルを
再検証し、64KiB の最初のチャンクを Rack 出力側へ渡した後に started を記録する。
HEAD や受付 POST では started にしない。確認には30秒の独立した期限がある。
サーバーの出力開始はブラウザー受信・保存完了を保証しない。途中の切断もあり得る。
開始確認前の失敗はダイアログに残る。再実行はダイアログを閉じてエクスポートをやり直す。
Web と worker は同じ COUNTS_EXPORT_DIRECTORY、Rails secret、DB、実行ユーザーで
共有 POSIX filesystem を使うこと。別ホストではローカル tmp を共有できない。
systemd の EnvironmentFile と Web の環境を照合し、directory の owner と0700相当の
アクセス制限、容量、期限切れ削除、worker heartbeat と restart を公開前に確認する。
この作業では systemctl・worker 起動・本番環境への適用を行っていない。
これらのコード対策と回帰テストを追加しているが、テスト・型チェック・build は未実行。
middleware/proxy 経由の転送は別途実環境で確認する。
## 開始確認の競合と権限
開始受付は pending、実 GET の単独取得は streaming、最初の Rack yield 後は started。
同じ job の重複受付は既存 attempt を返す。streaming への遷移はプロセス間ロックで
単独取得し、同一 attempt の二重 GET を拒否する。開始前の attempt は45秒で失効する。
Frontend は30秒の独立タイマーで確認を打ち切る。一時的なポーリング通信断は同じ
attempt で再取得するが、別の GET は始めない。取消失敗後の再試行は利用者の再操作で行う。
ファイル・ジョブは owner、署名、期限、現在の権限 revision 及び artifact_policy を
検証する。最初の送出前にも現ユーザー・BAN・ファイル権限を再確認する。旧ポリシーの
生成物は再利用しない。送出中は各チャンクで取消状態を確認し、例外時はログと failed
状態を記録して body を閉じる。コミット後に head で別レスポンスを書かない。
ResponseBody に to_path を提供せず、sendfile middleware による ACK の迂回を避ける。
Last-Modified と X-Accel-Buffering: no を設定するが、プロキシの動作まで保証しない。
サーバーの最初の Rack yield は、クライアントの受信開始の厳密な証明ではない。
非管理者の summary では、公開集計属性は操作日時・異動区分。その他の属性に対する
抽出条件・補助タグ検索は拒否する。measures / column_order はマスク後の式だけと
比較するため、推測した private ID は一致しない。予約値そのものの条件指定は拒否する。
private ID の正誤に応じて許可可否を変えないため、条件の一致探索を避けられる。
非公開列を含む列順変更は API の column_values_editable でも不可と通知する。
公開列の値を使うソートは引き続き利用できる。
明細には元の認可を適用し、抽出条件自体や取得済みの全言語値を表示設定で改変しない。
外部タグ履歴の linked_tag_ids は、連携する内部タグの ID を半角スペース区切りで
記録する。除票履歴のある内部タグの ID を含むスナップショットを非管理者へ公開しない。
名前や UI ロカールを参照 ID の判定に使わない。
外部タグ名は履歴に保存せず、external_tags の platform と name を参照する。
V1 Counts の公開フィールド名は linked_tags を維持し、DB の linked_tag_ids へ対応付ける。
明細・フィルタ・集計軸・変更項目・ソート・CSV / JSON 出力で同じ対応付けを使う。
既定列と既存の表示ラベルも linked_tags のままとし、値は ID のスペース区切りとする。
linked_tag_ids は物理カラム名であり、Counts の公開フィールドとしては受け付けない。
旧 URL や保存済み設定のフィールド指定は維持できるが、値にタグ名を指定した検索条件や
ドリル条件は ID 値への見直しが必要となる。旧履歴の名称は移行後の ID 列から復元できず、
現在の名称からの推定変換も行わない。
+148
ファイルの表示
@@ -0,0 +1,148 @@
# Counts リリース前確認
この作業では Git、テスト、lint、typecheck、build、migration、サーバー・worker・
systemd の起動、本番適用を行っていない。コードの静的確認とテストコードの編集のみ。
以下の実行と結果確認は利用者が行う。未実行の状態は GO としない。
## 対応箇所
- 全件集計と安全な軸:counts_axes.rb / counts_table_query.rb / counts_pivot_totals.rb。
非公開値は GROUP BY 前に統合。通常セル・ソート用集計・小計は同じ寄与母集合を使う。
- 出力:counts_table_export.rb / countsDisplay.ts / CountsSummaryResults.tsx。
CSV とコピーは伏字。JSON の非公開座標は予約値のみ。非公開セルはリンクを作らない。
- 配布:counts_exports_controller.rb / counts_export_jobs.rb / counts_export_download.rb。
owner/job/attempt、単独 GET、期限・現在権限・旧 artifact policy の拒否を確認する。
- 表示:summary 行番号を廃止し「行数」を表示。details の行番号・件数は維持。
非公開列座標のソート・並び替え可否はサーバーの能力フラグへ従う。
- テスト基盤:requestAnimationFrame の同期・時刻0固定を修正。JSDOM の Canvas は
計測非対応を返すテスト専用モックを置く。実ブラウザーの計測動作は変更していない。
## 利用者によるテスト
backend ディレクトリで提案する対象(未実行):
```sh
bundle exec rspec spec/requests/counts_table_spec.rb spec/requests/counts_download_spec.rb spec/services/counts_pivot_security_spec.rb spec/services/counts_export_spec.rb spec/models/history_access_spec.rb spec/requests/post_locale_filter_identity_spec.rb spec/requests/tag_description_wiki_spec.rb
```
frontend ディレクトリで提案する対象(未実行):
```sh
npm run test:run -- src/lib/counts.test.ts src/lib/api.test.ts src/pages/counts/CountsResourcePage.test.tsx src/pages/help/HelpChangesPage.test.tsx src/components/counts/CountsControls.test.tsx src/components/counts/CountsSummaryResults.test.tsx src/components/counts/CountsExportProgress.test.tsx
```
SSR は frontend/server.ssr.test.mjs が dist/server/entry-server.js を読み込む。
ソースの PageHead はサイト名付きの共通タイトルを使用している。古い成果物のまま
SSR テストを実行しても現行ソースの検証にならない。利用者が最新成果物を用意してから
`npm run test:ssr` を実行する。Codex はビルド・SSR テストとも実行していない。
主要テストの後に既存の全テスト群でも回帰の有無を確認する。
## ブラウザーと本番切替前
- anonymous / guest / member / admin で同じ期間・公開条件を指定し、通常セル・総計の
数値を比較する。安全なグループ数は権限で異なり得る。details は秘匿版を出さない。
- 投稿 A,B / A / B,C のタグ行軸は5。タグを行軸から外した合計は distinct 投稿数。
changed_field、NULL、複数段小計、ページ境界、CSV / JSON でも値を比較する。
- 伏字セル、API 辞書、DOM、コピー、CSV / JSON に非公開の値・ID がないこと。
非公開セルのドリルダウンはなく、private 条件の summary は安全な403となること。
- 8軸・100列の専用エラー、SQL timeout、worker 不在、認証・権限、取消失敗を確認する。
- CSV / JSON とも ready / 100% だけでは閉じず、実 GET の matching attempt が
started となってから閉じること。通信遮断・取消・再描画・タイムアウトも確認する。
サーバー送出開始はブラウザーの受信完了・保存完了を保証しない。
- PC・スマホで独立/共通スクロール、D&D、列幅、セル選択・文字列選択・キャレット、
内表 JSON の折返し、ページタイトル、Counts の noindex・先読み禁止を確認する。
locale 切替後も対象 ID と集計値は同一で、全言語の登録値を保持すること。
- Web と worker の同一リリース・secret・DB・COUNTS_EXPORT_DIRECTORY、所有者、
flock / atomic rename、容量、期限切れ掃除、heartbeat、ログ、再起動を照合する。
- systemd unit の User / WorkingDirectory / EnvironmentFile / PATH を実環境と照合する。
設定適用・サービス起動・本番切替はこの作業では行っていない。
- ロール変更・logout・新しい discard 後に、生成済みファイルが再取得できないこと。
- ロールバック時も公開ダウンロードを止め、旧生成物を再配布しない。古いコードへ戻すと
artifact_policy やマスクの保護を失う可能性があるため、無条件に巻き戻さない。
P0 の回帰テストと利用者の動作確認が済むまで判定は NO-GO。
## 静的な対応状況
| 対象 | 状態 | 根拠 |
| --- | --- | --- |
| 非公開軸・参照・全件母集団 | 対応済み(静的確認のみ) | HistoryAccess / CountsAxes / CountsTableQuery と回帰コード |
| タグ・変更項目・小計 | 対応済み(静的確認のみ) | CountsPivotTotals と全セル・ソートの母集合の統一 |
| API / CSV / JSON / コピー / URL | 対応済み(静的確認のみ) | マスク後の値のみを表示・比較し、予約座標の条件化を拒否 |
| GET ACK・競合・取消・期限 | 対応済み(静的確認のみ) | サーバー状態遷移、単独 GET、独立タイマー、回帰コード |
| P1・既知テストのソース上の問題 | 対応済み(静的確認のみ) | 行番号、行数、エラー、RAF、fixture、title 契約 |
| テスト成功・実ブラウザー・本番稼働 | 判定不能 | 実行禁止のため未検証 |
## 今回の編集ファイル
すべて Counts 本体に必要な変更。共有箇所の例外は HistoryAccess の参照秘匿と、
api.ts の Abort/CSRF 待機・native download の機構であり、全域の安全性にも必要。
SQL の速度改善・索引追加は別 Issue とし、今回 migration は追加していない。
### Backend App
- backend/app/models/history_access.rb:外部タグの記録名参照と、投稿の記録済み参照版にも
同じ可視性判定を適用。旧 artifact policy の生成物を拒否。
- backend/app/services/counts_population.rb:軸条件の型・予約値の検証。
- backend/app/services/counts_axes.rb:マスク前の内部寄与を計数し、直前版が非公開の場合も
changed_field の名前を秘匿。
- backend/app/services/counts_table_query.rb:通常セル・ソートの寄与母集合、整数順、
条件表示名、公開列座標だけの操作、summary メタデータを整合。
- backend/app/services/counts_pivot_totals.rb:全セルに寄与する母集合と、タグ行軸の加算。
- backend/app/services/counts_table_export.rb:CSV の予約値を伏字化。
- backend/app/services/counts_export_jobs.rb:開始確認の排他・状態遷移・失効と安全なエラーコード。
- backend/app/services/counts_export_download.rb:64KiB の Rack body、開始確認、取消・close。
- backend/app/controllers/counts_exports_controller.rb:GET 直前・送出直前の再認可と専用エラー。
### Frontend App
- frontend/src/lib/api.ts:CSRF 待機中も中断可能にし、iframe の寿命・失敗通知を整理。
- frontend/src/lib/counts.ts:マスク予約値と列操作の API 型。
- frontend/src/lib/countsDisplay.ts:マスク表示と既知エラーの i18n 解決。
- frontend/src/components/counts/CountsExportProgress.tsx:一つのジョブを維持し、matching ACK、
独立期限、取消・再描画・通信断を処理。保存完了の成功表示はしない。
- frontend/src/components/counts/CountsSummaryResults.tsx:行番号コードの撤去、私的セルの
ドリルダウン抑止、公開列のソート維持。
- frontend/src/components/counts/CountsSortHeader.tsx:禁止座標のソート抑止と伏字の理由表示。
- frontend/src/pages/counts/CountsResourcePage.tsx:専用エラーと、初回失敗後にも再実行できる状態。
- frontend/src/locales/ja/counts.json:承認されたエラー文言。
- frontend/src/locales/ain-Kana/counts.json:同じキーと未翻訳・日本語 fallback の管理。
### テスト・テスト専用基盤
- backend/spec/services/counts_pivot_security_spec.rb(新規):4権限、タグ寄与5、distinct、
changed_field、private group、NULL、整数順、ページ、座標探索、JSON/CSV。
- backend/spec/requests/counts_download_spec.rb(新規):CSV/JSON の実 GET、HEAD、二重取得、
別セッション、失効、取消、ファイル不在、新しい discard。
- backend/spec/services/counts_export_spec.rb:owner/job/attempt の ACK と Rack body の失敗。
- backend/spec/models/history_access_spec.rb:旧 policy、外部タグ参照、記録済みタグ参照版。
- backend/spec/requests/counts_table_spec.rb:専用軸数エラー。
- frontend/src/components/counts/CountsExportProgress.test.tsx(新規):matching ACK、StrictMode、
再描画、開始受付失敗、通信断、独立期限、取消の競合・再試行、unmount、iframe 失敗。
- frontend/src/components/counts/CountsSummaryResults.test.tsx(新規):伏字・理由・コピー・旧行番号。
- frontend/src/components/counts/CountsControls.test.tsx:summary 行番号廃止と表示件数選択。
- frontend/src/pages/counts/CountsResourcePage.test.tsx:初期エラーからの再実行、サイト名付き title。
- frontend/src/lib/api.test.ts:共有 CSRF 待機中の中断、iframe の解放・重複失敗通知。
- frontend/src/lib/counts.test.ts:理由別エラーと任意のサーバー文字列を表示しないこと。
- frontend/src/test/setup.ts:時刻が進む非同期 RAF と Canvas のテスト専用 fallback。
- frontend/server.ssr.test.mjs:共通 title の既存 assertion を維持し canonical と秘匿を補足。
### 文書
- docs/counts-exports.md:実装と一致する計数・ACK・権限・運用上の限界。
- docs/counts-release-checklist.md:この一覧と利用者の検証・公開判定手順。
## 既知失敗の静的な確認結果
- counts_table_spec.rb:TagVersion.names の auto_generated、script_code、aliases があることを
確認。新しい投稿タグ fixture にも schema 必須の sections を設定。
- post_locale_filter_identity_spec.rb:検索文字列で基準集合を取得した後、tag_filters の
応答から references を定義して構造化 URL を作る修正を維持。
- tag_description_wiki_spec.rb:first / last とも対象 ID に限定した delete_all を維持。
- CountsResourcePage.test.tsx / HelpChangesPage.test.tsx:共有 RAF の同期・時刻0固定を修正。
Help の辞書応答は実 getHelpChanges が language を付ける契約も照合済み。
- SSR title:現行ソースはサイト名を含む。dist を使うテストの assertion は削除していない。
上記はソース照合の結果であり、過去の失敗件数が0になったという実行結果ではない。
既知の未修正 P0 を残して GO とする判断はしていない。現時点の阻害事項は、ユーザーに
よる MySQL・ブラウザー・worker/proxy を含む実検証がまだ行われていないことである。
+247
ファイルの表示
@@ -0,0 +1,247 @@
# Static Mirror PoC (#483)
現在の公開状態の一部を、Rails で HTML にして ZIP 出力する開発環境専用の PoC。
Post は ID・タイトル・URL・出力済み Tag との関連、Tag は ID・カテゴリ・全言語の
名前と別名・出力済み Post との関連を出力する。それ以外の Post / Tag 属性は対象外。
DB backup / restore とは別機能で、日時指定の復元機能はない。
本番環境では全 API が 404 になる。依存パッケージ・DB schema の変更はない。
## 操作
既存のローカル開発用 MySQL 設定と locale 登録を用意した環境で、
`backend/` の `bin/rails server -p 3002` と、`frontend/` の `npm run dev` を使う。
Vite の既存 `/api` proxy は3002番ポートを参照する。これらの起動コマンドは
今回実行していない。DB の準備・migration・依存導入も実行していない。
1. `http://localhost:5173/ja/static-mirror` を開く。開発時の「その他」にもリンクがある。
有効な locale では `/:locale/static-mirror` から利用できる。
2. 「スナップショット生成」を押す。ログインは不要。通常の CSRF、IP BAN、user BAN
の境界を通り、セッションに紐づくジョブを登録する。
3. 状態・処理件数・データ基準時点を確認し、完了後「ZIP ダウンロード」を押す。
4. ZIP を展開し、`archive/` を通常の静的 HTTP サーバーへ配置する。
`/保存版/archive/index.html` のようなサブディレクトリ配置に対応する。
DB、Node.js、原サーバー、外部 API は閲覧先では不要。
一覧・詳細・相互リンクは JavaScript 無効でも利用できる。検索は HTTP 配信した
`search-index.json` を JavaScript が読む。`file://` からの検索は保証しない。
Post はタイトルに加えて、出力対象の関連 Tag の全言語名・別名から検索できる。
非公開・廃止・件数上限外の Tag や、関連しない Tag の名前は Post の検索項目に含めない。
外部動画・画像・サムネール・素材ファイルは取得も同梱もしない。Post の外部 URL は
通常のリンクとして残すため、そのリンク先の閲覧は外部サイトの存続に依存する。
各 HTML に静的保存版の表示、UTC のデータ基準時点、`noindex, nofollow`、
本家への絶対リンクを焼き込む。内部リンク・CSS・JS は相対参照。
本家リンクの初期オリジンは `https://hub.nizika.monster`。request の Host、
forwarded header、API のオリジンは採用しない。
`archive-config.js` の `window.archiveConfig.origin` を HTTP(S) の本家基点 URL
(例: `https://example.org`)に変更すれば本家リンクをまとめて変更できる。
設定 JS が削除・欠落・不正な場合や JS 無効時にも HTML の初期リンクが残る。
設定は配置先ミラーの URL ではなく、本家の基点を指す。
## API と処理上限
| API | 役割 |
| --- | --- |
| `GET /static_mirror` | 件数・容量・実行制限と対象外機能 |
| `POST /static_mirror/jobs` | 現在状態のジョブ登録。入力は `locale` のみ |
| `GET /static_mirror/jobs/:id` | 所有セッションの状態・処理進捗・基準時点・失敗コード |
| `GET /static_mirror/files/:id` | 所有セッションの完成 ZIP を再認可して取得 |
通常どおりフロントエンドの API transport と Query key を使用する。
URL に JSON を格納しない。`at` 等の任意時点入力・未知の生成オプションは400で拒否する。
- Post / Tag は各100件。公開認可後に ID 昇順で選ぶ。非公開件数や全母集団件数は返さない。
- 対象 Tag の全言語名・別名は最大2,000行。超過時は失敗させ、黙って切り捨てない。
言語・文字体系は保持し、表示対象言語に登録がない場合は ID を使う。
- Post と Tag の関連は両者が出力集合にあるものだけ。出力外への内部リンクは生成しない。
- 展開後のファイル合計及び ZIP は各8 MiB 以下。超過時は配布しない。
- 同時実行1件、全セッションを通じて新規登録は60秒に1件、登録から処理期限60秒。
競合・受付制限は429と `Retry-After`。状態取得・取得時の短いロック競合は503。
- ZIP とジョブの認可期限は10分。次のジョブ登録で前の ZIP / ジョブは置き換える。
queued / running の予約は cooldown が過ぎても置き換えない。75秒を超えた予約は
状態取得又は次の受付で failed にし、実行ロックが空いている場合だけ次を受け付ける。
期限後も最後のファイルがディスクに残り得るが、API では取得できない。
容量は単一 ZIP と作業ファイルに限定され、無制限に蓄積しない。
Rails の既存 Active Job と `:async` adapter を使い、生成は HTTP request の外で行う。
既存 `CountsExportJobs` は Counts の入力・投影・配信確認に結合しているため変更せず、
本機能の状態・単一成果物の管理だけを feature 固有の Job に置く。
受付・状態・ZIP は `backend/tmp/static_mirror/` の単一 spool を使う。
プロセス間の filesystem lock で実行枠と更新を制御する。各ファイルは0600、directory は
0700で作成する。Web プロセス再起動で非永続キューが失われる点が PoC の限界。
進まない queued / running は75秒経過後の状態取得又は次の受付で failed になる。
worker は受付中の短い実行ロック競合を待ち、競合だけを理由に queued を放棄しない。
再開や再実行はしない。実行ロック待ちも登録からの60秒の処理期限に含まれる。
停滞検知で failed になったジョブは、遅れた worker の進捗・完了・例外処理から
書き換えない。queued の読み取り直後に停滞判定された場合も、生成を開始しない。
失敗コードは `too_large`、`generation_timeout`、`publication_changed`、
`generation_failed`、`worker_stopped`。内部の例外本文・パス・資源属性は応答に含めない。
画面では失敗コード又は既存の通信・認可エラー文言を表示する。
## 公開制御と整合性
生成者の role によらず `HistoryAccess.set(user: nil)` の公開権限を使う。
既存の `HistoryAccess.current` と確定済みの同一 `version_no` の存在確認を組み合わせる。
独自の role 判定や簡易な除票判定は作らない。対応する immutable revision がない
legacy record は安全側で対象外とする。公開不可の参照を含む確定版も既存認可で除外する。
廃止 Tag は対象外。HTML / 検索 JSON / manifest の件数には認可済みの出力集合だけが入る。
user、actor、履歴版番号、監査、認証、blob 情報は投影しない。
公開条件の調査範囲は V1 の Post / Tag / TagName / Locale の model、schema、
閲覧 controller・representation、HistoryAccess と Material の抑止経路。
Post / Tag / TagName には別の private / visibility カラムや、汎用の公開抑止 scope は
見つからなかった。Tag の `deprecated_at` は除票とは別の廃止状態で、通常の Post
表示からも除外されるため、ミラーから引き続き除外する。Locale の廃止後も配布しない。
`MaterialImportBlock`、`MaterialSyncSuppression` は著作権・個人情報等を理由にした
素材取込・同期の抑止であり、Post / Tag の汎用認可として流用しない。Material と
関連ファイル・監査用の理由・同期元情報は全て出力対象外のままにする。
既存の通常 Post / Tag 閲覧経路が共通認可を一律適用しているとは仮定せず、
ミラー自身は必ず HistoryAccess を経由する。新たな抑止条件を導入する場合には
適用範囲を確認して共通認可と静的配布契約を更新する必要がある。
`CountsExporter.with_read_snapshot` の既存 repeatable-read 境界を再利用し、Post、Tag、
全言語 TagName、PostTag、publication revision を同じ読み取り view から取得する。
これには既存 HistoryAccess の SQL が動作する MySQL 環境が必要。未対応の adapter や
未整備の DB では生成失敗とし、認可を迂回した出力へフォールバックしない。
`data_at` は DB 読み取り transaction を要求する直前の、取得処理開始時の
サーバー時計(UTC)。manifest の `data_at_basis: capture_started_at` に意味を記録する。
MySQL の read view は後の最初の consistent read で成立するため、その厳密な成立時刻や
`data_at` 以前のコミットだけを含むことは保証しない。履歴の指定日時復元も表さない。
独立した worker では repeatable-read を開始するが、呼出元の transaction がある場合は
既存 helper の契約どおり、その transaction の view と分離レベルを引き継ぐ。
`generated_at` はファイル生成後の別の時刻。
静的配布契約は `StaticMirror::PUBLICATION_POLICY` の
`static-mirror-public-projection-1`。Counts 用 `HistoryAccess::ARTIFACT_POLICY`、
`authority`、`authorised_artifact?` は使わない。共通 HistoryAccess の現在版認可と
全後続除票の revision 検知は再利用し、集計件数の例外や admin の配布モードは設けない。
旧 Counts 契約の metadata しかない ZIP は再生成を必要とする。
生成の最後と状態・ZIP 取得時に、別の読み取り transaction で共通 discard revision と
出力対象 ID の公開投影を再取得する。SQL query cache を使わず、対象の存在、確定版、
Tag / Locale の廃止、タイトル・URL・category、全言語名・別名・文字体系、関連の変更を
検知する。除票を伴わない訂正・削除も投影 digest の不一致として配布を止める。
対象 ID と digest はサーバー側ジョブ metadata にだけ置き、manifest や API 状態応答へ
出さない。対象は各100件・名前2,000行以内の一括取得で、行ごとの追加 query は行わない。
生成中の除票は失敗にし、生成後の除票では古い ZIP の再取得を404にする。
ZIP は再認可後のバイト列として返し、遅延 sendfile に古いパスを渡さない。
HTTP 応答は既存境界の `private, no-store`。ダウンロード済み ZIP の回収や、
最終認可と送出の間に発生する公開制御変更の完全な排他は保証できない。
新しい公開制御を導入する場合は共通 policy と静的配布契約の連動が必要。
本文・名前・URL は escape する。外部リンクは HTTP(S)・host・userinfo 不在を検証し、
不正な URL はリンクとして出さない。外部 URL の取得は行わないため SSRF 用 DNS 検査を
流用しない。HTML の CSP は外部スクリプト・画像・通信を許可しない。
`manifest.json` に形式版、生成日時、基準時点、locale、出力件数、対象外項目、上限、
各ファイルの相対パスと SHA-256 を記録する。自己ハッシュは成立しないため manifest
自身だけは一覧の対象外とし、`checksum_scope` にその範囲を記録する。
## 過去時点復元の調査
- `post_versions` / `tag_versions` は親 ID と正の `version_no` に一意制約があり、
readonly の `VersionRecord` を使う。`created_at <= 時点` の中で最大の確定版を
選ぶことは可能。同一時刻の順序は時刻だけで決めず版番号を併用する必要がある。
- Tag は `names` に言語別の名前・script・別名を保存する。古い行の移行では
日本語として backfill し、旧 aliases 文字列を分割している。失われた区切りや
当時未記録の他言語名を現在の TagName で補完して正確と称することはできない。
- Post は `tags_json`、区間、Tag 版参照、`parent_post_ids` を、Tag は
`parent_tag_ids` を記録する。ただし legacy `tags_json` は名前・時刻からの補完を
含む。全時代・全書き込み経路の関係変更の完全性を検証する必要がある。
外部 Tag 名の現行参照など、immutable history だけでは揃わない値もある。
- Tag Description は `(tag_id, language_code, version_no)` の版に本文と翻訳元を記録する。
Material にはファイル情報等の版があるが、blob 本体の永久保存や除票・公開抑止との
整合は別途必要。Help はフロントエンドの文書配信に依存し、DB の任意時点だけでは
当時の公開版を復元できない。
- 当時の publication policy そのものや将来追加される公開抑止の完全な履歴は、
これらの内容版だけでは保証できない。指定時点の再現と現在の公開許可を別々に評価し、
現在の全後続除票境界を版番号と複合キーで適用する必要がある。恢復しても過去の
秘匿履歴を再公開しない。Counts の件数例外はこの機能へ拡張しない。
- 正確な時点復元には、全 resource の履歴の起点・欠落・backfill の識別、関係・名前・
policy・locale・asset の時点整合性、immutable な Help の版、削除済み asset の保存方針、
一つの DB read view と現在の認可の再検証、同時刻更新の順序・境界の設計が必要。
このため日時入力は実装していない。履歴評価はソース調査であり、実 DB の網羅監査ではない。
## 検証と残課題
追加した回帰テスト:
- `backend/spec/services/static_mirror_spec.rb`: ZIP の CRC / 構造、全ファイルの hash、
内部参照、escape、基準時点、公開権限、後続除票、言語名・別名、容量・件数上限。
今回は Counts 契約との独立性、除票なしの訂正・廃止・関連変更、取得開始時刻の意味、
Post 検索への関連名の投影を追加。
- `backend/spec/requests/static_mirrors_spec.rb`: 一般受付、非同期境界、所有権、
公開制御変更、失敗・期限・競合・全体 rate limit、開発環境制限、CSRF と BAN。
今回は Host に依存しない本家リンク、訂正後の取得拒否、queued 予約の保持・期限切れ、
実 filesystem lock を使う受付競合待ちを追加。
- `frontend/src/pages/StaticMirrorPage.test.tsx`: 開始、進捗、取得、失敗、
受付エラー、認可喪失、日本語 fallback の言語属性。
- `frontend/src/lib/staticMirrorAssets.test.ts`: 静的検索の相対参照・escape と
サブディレクトリ配備、本家設定の欠落・不正・変更。
今回は Tag 名・別名から関連 Post が見つかるケースを追加。検索 JS の変更は不要。
実施したのは既存コードの調査、差分の目視確認と `git diff --check`(成功)。
テスト、lint、build、型検査、
migration、実ブラウザーでの表示、ZIP 展開・HTTP 配置は実行していない。
管理者指定の suite による結果確認と、サブディレクトリ配置・JS 無効・原サーバー停止・
設定 JS 欠落・静的検索の実画面確認が残る。
本番公開には durable な worker / queue、復旧・取消・削除と運用監視、適切な利用者別
rate limit、容量・時間・負荷の実測、公開制御の監査と変更検知、長時間配信の再認可方針、
asset の権利・保管方針、追加 resource の安全な投影、UI 文言・翻訳の確定が必要。
本 PoC の開発環境制限を単に外して公開することは想定しない。
## 追加点検での変更ファイル
レビュー5点の既存実装を確認し、その実装を維持した。追加修正は単一生成枠の状態遷移に
限定した。従来の `update` は ID の一致だけを確認していたため、状態取得又は次の受付が
停滞ジョブを failed にした後も、worker が running / ready へ戻す競合が残っていた。
registry lock の中で failed 状態の更新を拒否し、生成開始の更新を拒否された worker は
そのまま終了する。生成中に停滞判定された場合も、進捗・完了・失敗理由の上書きを防ぐ。
| ファイル | 修正と理由 |
| --- | --- |
| `backend/app/jobs/static_mirror_job.rb` | failed への遷移を worker が取り消さないよう、更新と生成開始を制限 |
| `backend/spec/requests/static_mirrors_spec.rb` | queued 読取直後の停滞検知、running 失効後の進捗・完了と ZIP 取得拒否を検証する2ケースを追加 |
| `docs/static-mirror.md` | 今回の競合修正と未検証範囲を記録 |
新規回帰テストは、状態取得と worker 更新の順番を制御して競合を再現する。
テスト・lint・build・migration、DB 接続、実ブラウザー・ZIP 配置確認は実行していない。
非永続キュー、ロック取得自体の無期限待ち、厳密な DB read view 成立時刻の取得、
過去時点復元、本番公開に必要な検証と運用設計は未解決のまま。
公開契約・検索・本家リンク・`data_at` の意味及び機能範囲に追加変更はない。
## 前回までのレビュー対応の変更ファイル
| ファイル | 修正と理由 |
| --- | --- |
| `backend/app/controllers/static_mirrors_controller.rb` | request オリジンを生成処理へ渡さず、本家リンクへの Host 混入を防止 |
| `backend/app/jobs/static_mirror_job.rb` | 静的配布専用の再認可、queued 予約と期限切れの共通処理、worker のロック待ちでジョブ消失を防止 |
| `backend/app/services/static_mirror.rb` | Counts と独立した公開投影の再検証、関連 Tag 名を含む Post 検索、正規の本家リンク初期値、`data_at_basis` |
| `backend/spec/services/static_mirror_spec.rb` | 専用契約・訂正・廃止・関連変更・検索投影・時刻意味の回帰テスト |
| `backend/spec/requests/static_mirrors_spec.rb` | 本家オリジン・ZIP 再認可・queued 予約・実行ロック競合の回帰テスト |
| `frontend/src/lib/staticMirrorAssets.test.ts` | 名前・別名を使う Post 検索の回帰テスト |
| `docs/static-mirror.md` | 公開条件の調査結果、配布契約、ロックと時刻の保証範囲・未解決事項を更新 |
V2、DB schema / migration、既存の Counts 契約・共通認可、UI 文言は変更していない。
## 変更ファイルの責務
| ファイル | 責務 |
| --- | --- |
| `backend/config/routes.rb` | metadata・job・ZIP の4経路 |
| `backend/app/controllers/static_mirrors_controller.rb` | 通常認証境界、開発制限、入力・HTTP 応答 |
| `backend/app/jobs/static_mirror_job.rb` | 単一生成枠、受付制限、状態・成果物・期限・取得認可 |
| `backend/app/services/static_mirror.rb` | 公開データの一貫した投影、ページ・検索・manifest・ZIP |
| `backend/app/views/static_mirror/page.html.erb` | escape 済み HTML、ナビ、基準時点・本家リンク・CSP |
| `backend/app/views/static_mirror/archive.css` | 静的サイトの読み取り表示 |
| `backend/app/views/static_mirror/archive.js` | ローカル検索と本家リンクの設定変更 |
| `frontend/src/App.tsx` | locale 付き操作画面経路 |
| `frontend/src/lib/queryKeys.ts` | 既存 Query key entrypoint への追加 |
| `frontend/src/pages/StaticMirrorPage.tsx` | 開始・制限・進捗・失敗・ZIP 操作 |
| `frontend/src/pages/MorePage.tsx` | 開発時の画面への導線 |
| `frontend/src/locales/{ja,ain-Kana}/archive.json` | Issue 指定の文言と未翻訳 fallback |
| 上記4テストファイル | 回帰テスト |
| `docs/static-mirror.md` | 手順・制約・調査・未検証事項 |
+57 -19
ファイルの表示
@@ -62,14 +62,37 @@ const responseFor = url => {
return { locales: [{ code: 'ja', name: '日本語' }, { code: 'ain-Kana', name: 'アイヌ' }] }
if (path === '/users/settings') return { theme: 'system', wiki_editor_mode: 'split' }
if (path === '/users/theme_slots') return []
if (path === '/counts/posts' || path === '/counts/tags') {
const resource = path.endsWith ('posts') ? 'posts' : 'tags'
return { count: 1, page: 1, limit: 20, group_count: 1,
groups: [{ key: url.searchParams.get ('axis') === 'user' ? 1 : '2024-01',
name: null, count: 1 }],
versions: resource === 'posts'
? [{ post_id: 1, version_no: 1, title: { current: `SSR count ${locale}` } }]
: [{ tag_id: 1, version_no: 1, name: { current: `SSR count ${locale}` } }] }
if (path.startsWith ('/counts/definitions/')) {
const resource = path.split ('/')[3]
const language = locale === 'ain-Kana' ? 'ain' : 'ja'
const names = ['ja', 'ain'].map (code => ({ key: `name_${code}`,
label: 'counts:localeName', name_locale: { code, language_code: code } }))
const columns = resource === 'posts' ? ['resource_id', 'title']
: ['resource_id', `name_${language}`]
return { fields: [{ key: 'resource_id', type: 'integer', nullable: false,
operators: ['eq', 'ne'] }],
output_fields: [{ key: 'resource_id', label: 'counts:columns.resourceId' },
...(resource === 'posts' ? [{ key: 'title', label: 'counts:columns.postTitle' }] : names)],
name_columns: names, filter_fields: [], column_aliases: [], condition_aliases: [],
axes: { latest: ['resource_id'], history: ['resource_id'] }, grains: ['month'],
defaults: [
{ definition: 'latest', mode: 'summary', params: [
{ key: 'rows', value: '' }, { key: 'cols', value: '' }] },
...['latest', 'history'].map (definition => ({ definition, mode: 'details', params: [
{ key: 'columns', value: columns.join (' ') },
{ key: 'sort', value: 'resource_id:asc version_no:asc' }] }))] }
}
if (path === '/counts/query') {
const header = { values: [], kind: 'data', depth: 0, axis_count: 0, label_key: null }
return { count: 1, page: 1, limit: 100, group_count: 1,
output_count: 1, visible_output_count: 1, excluded_count: 0,
excluded_count_scope: 'current_query', execution_at: date,
rows: [], cols: [], row_headers: [header], column_headers: [header],
column_values: [[]], cells: [{ values: [], count: 1 }], pivot_totals: [],
condition_labels: {}, axis_labels: {}, versions: [{ id: 1, resource_id: 1,
version_no: 1, latest: true, created_at: date, created_by_user: null,
title: 'SSR count', name_ja: 'SSR count', name_ain: 'SSR count',
names: [], aliases: [], parents: [], tags: [] }] }
}
if (path === '/posts') return { posts: [post (locale)], count: 1 }
if (path === '/gekanator/posts') return { posts: [post (locale)] }
@@ -309,21 +332,36 @@ after (async () => {
if (errors.length) throw new AggregateError (errors, 'SSR teardown failed')
})
test ('counts SSR renders history data and retains URL query state', async () => {
test ('counts SSR renders the shell without result requests or embedded results', async () => {
for (const locale of ['ja', 'ain-Kana']) {
for (const resource of ['posts', 'tags']) {
const column = resource === 'posts' ? 'title' : 'name'
const query = `mode=details&columns=${column}&user_id=4`
for (const [resource, mode] of ['posts', 'tags'].flatMap (resource =>
['summary', 'details'].map (mode => [resource, mode]))) {
const column = resource === 'posts' ? 'title' : (locale === 'ja' ? 'name_ja' : 'name_ain')
const query =
`definition=history&mode=${mode}&columns=resource_id+${ column }&user_id=4&timezone=UTC`
const start = requests.length
const result = await renderPage (`/${locale}/counts/${resource}?${query}`, origin, apiURL)
const calls = requests.slice (start).map (request => new URL (request.url, origin))
assert.equal (result.status, 200)
assert.match (result.html, new RegExp (`SSR count ${locale}`))
assert.match (result.html, new RegExp (`/${locale}/${resource}/1`))
assert.doesNotMatch (result.html, /SSR count|counts-grid|counts-nested/)
assert.match (result.html, /counts-targets/)
assert.match (result.html, /<form[\s>]/)
assert.doesNotMatch (result.html, /counts:|columns\.changedAt|fields\.actor/)
assert.match (result.html, /aria-expanded="false"/)
if (locale === 'ain-Kana')
assert.match (result.html, /lang="ja"[^>]*>集計実行/)
assert.ok (requests.some (request => request.url.includes (`/counts/${resource}`)
&& request.url.includes ('user_id=4') && request.url.includes (`locale=${locale}`)))
assert.match (result.head, /<meta[^>]*name="robots"[^>]*content="noindex"/)
const titles = {
ja: { posts: '広場集計', tags: 'タグ集計' },
'ain-Kana': { posts: 'ミンタㇻ ウコピㇱキ', tags: 'シロシ ウコピㇱキ' } }
assert.match (result.head, new RegExp (
`<title[^>]*>${titles[locale][resource]} \\|[^<]*<\/title>`))
assert.match (result.head, /rel="canonical"/)
assert.doesNotMatch (result.html, /private-discard|hidden-title|__counts_private__/)
assert.ok (calls.some (url => url.pathname === `/counts/definitions/${resource}`))
assert.equal (calls.filter (url => url.pathname === '/counts/query').length, 0)
assert.doesNotMatch (result.bootstrap, /SSR count|conditionLabels|pivotTotals/)
const bootstrap = JSON.parse (result.bootstrap)
assert.equal (bootstrap.state.queries.filter (query =>
query.queryKey[0] === 'counts' && query.queryKey[1] === resource).length, 0)
}
}
})
+11 -6
ファイルの表示
@@ -81,8 +81,9 @@ const MaterialListPage = lazyPage (() => import ('@/pages/materials/MaterialList
const MaterialNewPage = lazyPage (() => import ('@/pages/materials/MaterialNewPage'))
const MaterialSyncSuppressionsPage = lazyPage (
() => import ('@/pages/materials/MaterialSyncSuppressionsPage'))
const CountsResourcePage = lazyPage (() => import ('@/pages/CountsResourcePage'))
const CountsResourcePage = lazyPage (() => import ('@/pages/counts/CountsResourcePage'))
const MorePage = lazyPage (() => import ('@/pages/MorePage'))
const StaticMirrorPage = lazyPage (() => import ('@/pages/StaticMirrorPage'))
const GekanatorUnavailablePage = lazyPage (() => import ('@/pages/GekanatorUnavailablePage'))
const NicoTagListPage = lazyPage (() => import ('@/pages/tags/NicoTagListPage'))
const TOSPage = lazyPage (() => import ('@/pages/TOSPage'))
@@ -126,6 +127,8 @@ export const pageRoutes = (
user: User | null,
setUser: Dispatch<SetStateAction<User | null>>,
): PageRoute[] => [
{ path: 'static-mirror', element: <StaticMirrorPage/>,
handle: { preload: StaticMirrorPage.preload, robots: 'noindex' } },
{ path: '', element: <LocaleRedirect to="/posts"/>,
handle: { redirectTo: '/posts' } },
{ path: 'posts', element: <PostListPage/>,
@@ -200,16 +203,18 @@ export const pageRoutes = (
handle: { locale: 'ja', sitemap: ['static'], preload: locale => locale === 'ja'
? GekanatorPage.preload () : GekanatorUnavailablePage.preload () } },
{ path: 'counts', element: <CountsResourcePage resource="posts"/>,
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex',
switchLocale: switchPostFilterLocale } },
{ path: 'counts/posts', element: <CountsResourcePage resource="posts"/>,
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex',
switchLocale: switchPostFilterLocale } },
{ path: 'counts/tags', element: <CountsResourcePage resource="tags"/>,
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
{ path: 'counts/nico_tags', element: <CountsResourcePage resource="nico_tags"/>,
{ path: 'counts/nico-tags', element: <CountsResourcePage resource="nico_tags"/>,
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
{ path: 'counts/tag_descriptions', element: <CountsResourcePage resource="tag_descriptions"/>,
{ path: 'counts/tag-descriptions', element: <CountsResourcePage resource="tag_descriptions"/>,
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
{ path: 'counts/tag_groups', element: <CountsResourcePage resource="tag_groups"/>,
{ path: 'counts/tag-groups', element: <CountsResourcePage resource="tag_groups"/>,
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
{ path: 'counts/materials', element: <CountsResourcePage resource="materials"/>,
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
+120
ファイルの表示
@@ -0,0 +1,120 @@
import { fireEvent, screen } from '@testing-library/react'
import userEvent from '@testing-library/user-event'
import { useLocation } from 'react-router-dom'
import { beforeEach, describe, expect, it, vi } from 'vitest'
import TagSearch from '@/components/TagSearch'
import * as api from '@/lib/api'
import { buildTag } from '@/test/factories'
import { renderWithProviders } from '@/test/render'
const LocationOutput = () => {
const location = useLocation ()
return <output data-testid="location">{location.pathname}{location.search}</output>
}
const renderSearch = () => renderWithProviders (
<>
<TagSearch/>
<LocationOutput/>
</>, { route: '/ja/posts?page=3' })
describe ('TagSearch', () => {
beforeEach (() => {
vi.spyOn (api, 'apiGet').mockResolvedValue ([])
})
it ('searches through native submission without an Enter key event', () => {
renderSearch ()
const input = screen.getByRole<HTMLInputElement> ('textbox')
fireEvent.change (input, { target: { value: '虹夏 not:TEST type:or' } })
expect (input.form).toBeInstanceOf (HTMLFormElement)
fireEvent.submit (input.form!)
const query = new URLSearchParams ({ tags: '虹夏 not:test', match: 'any' })
expect (screen.getByTestId ('location')).toHaveTextContent (
`/ja/posts?${ query }`)
expect (input).toHaveValue ('虹夏 not:test type:or')
})
it ('requests a search action from the software keyboard', () => {
renderSearch ()
expect (screen.getByRole ('textbox')).toHaveAttribute ('enterkeyhint', 'search')
})
it ('searches through the form when Enter is pressed', async () => {
const user = userEvent.setup ()
renderSearch ()
await user.type (screen.getByRole ('textbox'), '虹夏{Enter}')
const query = new URLSearchParams ({ tags: '虹夏', match: 'all' })
expect (screen.getByTestId ('location')).toHaveTextContent (
`/ja/posts?${ query }`)
})
it.each ([
{ label: 'composition flag', composing: true, keyCode: 13 },
{ label: 'legacy IME code', composing: false, keyCode: 229 }]) (
'cancels confirmation Enter with $label', ({ composing, keyCode }) => {
renderSearch ()
const input = screen.getByRole ('textbox')
fireEvent.change (input, { target: { value: '虹夏' } })
const uncancelled = fireEvent.keyDown (input, {
key: 'Enter', isComposing: composing, keyCode })
expect (uncancelled).toBe (false)
expect (screen.getByTestId ('location')).toHaveTextContent ('/ja/posts?page=3')
expect (input).toHaveValue ('虹夏')
})
it ('cancels Enter throughout composition even without keyboard IME flags', () => {
renderSearch ()
const input = screen.getByRole ('textbox')
fireEvent.compositionStart (input)
fireEvent.change (input, { target: { value: '虹夏' } })
const uncancelled = fireEvent.keyDown (input, {
key: 'Enter', isComposing: false, keyCode: 13 })
expect (uncancelled).toBe (false)
expect (screen.getByTestId ('location')).toHaveTextContent ('/ja/posts?page=3')
expect (input).toHaveValue ('虹夏')
})
it ('blocks direct submission during composition and allows it after confirmation', () => {
renderSearch ()
const input = screen.getByRole<HTMLInputElement> ('textbox')
fireEvent.compositionStart (input)
fireEvent.change (input, { target: { value: '虹夏' } })
fireEvent.submit (input.form!)
expect (screen.getByTestId ('location')).toHaveTextContent ('/ja/posts?page=3')
expect (input).toHaveValue ('虹夏')
fireEvent.compositionEnd (input)
fireEvent.submit (input.form!)
const query = new URLSearchParams ({ tags: '虹夏', match: 'all' })
expect (screen.getByTestId ('location')).toHaveTextContent (
`/ja/posts?${ query }`)
})
it ('selects the active suggestion without submitting the form', async () => {
vi.mocked (api.apiGet).mockResolvedValue ([buildTag ({ name: '虹夏' })])
const user = userEvent.setup ()
renderSearch ()
await user.type (screen.getByRole ('textbox'), '虹')
await screen.findByText ('虹夏')
await user.keyboard ('{ArrowDown}{Enter}')
expect (screen.getByRole ('textbox')).toHaveValue ('虹夏 ')
expect (screen.getByTestId ('location')).toHaveTextContent ('/ja/posts?page=3')
expect (screen.queryByText ('虹夏')).not.toBeInTheDocument ()
})
})
+44 -24
ファイルの表示
@@ -1,6 +1,6 @@
// TODO: タグ入力系すべてに同様の処理あるため共通化する.
import { useEffect, useState } from 'react'
import { useEffect, useRef, useState } from 'react'
import { useLocation } from 'react-router-dom'
import { useLocale, useLocaleNavigate } from '@/lib/useLocale'
@@ -10,7 +10,7 @@ import { inputClass } from '@/lib/utils'
import TagSearchBox from './TagSearchBox'
import type { ChangeEvent, FC, KeyboardEvent } from 'react'
import type { ChangeEvent, FC, FormEvent, KeyboardEvent } from 'react'
import type { Tag } from '@/types'
@@ -20,6 +20,7 @@ const TagSearch: FC = () => {
const locale = useLocale ()
const location = useLocation ()
const navigate = useLocaleNavigate ()
const compositionFlgRef = useRef (false)
const [activeIndex, setActiveIndex] = useState (-1)
const [search, setSearch] = useState ('')
@@ -74,7 +75,34 @@ const TagSearch: FC = () => {
setSuggestionsVsbl (false)
}, [locale])
const handleSubmit = (ev: FormEvent<HTMLFormElement>) => {
ev.preventDefault ()
if (compositionFlgRef.current)
return
const parts = search.split (' ')
const match =
parts.map (t => t.toLowerCase ()).includes ('type:or') ? 'any' : 'all'
navigate (`/posts?${
(new URLSearchParams ({
tags: parts.map (t => t.toLowerCase ())
.filter (t => t !== 'type:or')
.join (' '),
match })).toString () }`)
setSuggestionsVsbl (false)
setRequest ({ query: '' })
}
const handleKeyDown = (ev: KeyboardEvent<HTMLInputElement>) => {
if (compositionFlgRef.current
|| ev.nativeEvent.isComposing
|| ev.nativeEvent.keyCode === 229)
{
if (ev.key === 'Enter')
ev.preventDefault ()
return
}
switch (ev.key)
{
case 'ArrowDown':
@@ -90,13 +118,12 @@ const TagSearch: FC = () => {
break
case 'Enter':
if (activeIndex < 0)
break
ev.preventDefault ()
{
const selected = suggestions[activeIndex]
if (selected)
handleTagSelect (selected)
const selected = suggestionsVsbl ? suggestions[activeIndex] : null
if (selected == null)
break
ev.preventDefault ()
handleTagSelect (selected)
}
break
@@ -105,20 +132,6 @@ const TagSearch: FC = () => {
setSuggestionsVsbl (false)
break
}
if (ev.key === 'Enter' && (!(suggestionsVsbl) || activeIndex < 0))
{
const parts = search.split (' ')
const match = parts.map (t => t.toLowerCase ()).includes ('type:or') ? 'any' : 'all'
navigate (`/posts?${
(new URLSearchParams ({
tags: parts.map (t => t.toLowerCase ())
.filter (t => t !== 'type:or')
.join (' '),
match }))
.toString () }`)
setSuggestionsVsbl (false)
setRequest ({ query: '' })
}
}
const handleTagSelect = (tag: Tag) => {
@@ -144,8 +157,9 @@ const TagSearch: FC = () => {
}, [location.search])
return (
<div className="relative w-full">
<form onSubmit={handleSubmit} className="relative w-full">
<input type="text"
enterKeyHint="search"
data-shortcut-focus-search="true"
placeholder={message ('labels.tagSearch').text}
value={search}
@@ -153,12 +167,18 @@ const TagSearch: FC = () => {
onFocus={() => setSuggestionsVsbl (true)}
onBlur={() => setSuggestionsVsbl (false)}
onKeyDown={handleKeyDown}
onCompositionStart={() => {
compositionFlgRef.current = true
}}
onCompositionEnd={() => {
compositionFlgRef.current = false
}}
className={inputClass (false,
'px-3 py-2 dark:border-gray-600 dark:bg-gray-800 dark:text-white')}/>
<TagSearchBox suggestions={suggestionsVsbl && suggestions.length ? suggestions : [] as Tag[]}
activeIndex={activeIndex}
onSelect={handleTagSelect}/>
</div>)
</form>)
}
export default TagSearch
+38 -19
ファイルの表示
@@ -4,6 +4,7 @@ import { useLocation } from 'react-router-dom'
import { useLocale, useLocaleNavigate } from '@/lib/useLocale'
import { useUITranslation } from '@/lib/useUITranslation'
import PrefetchLink from '@/components/PrefetchLink'
import TagLink from '@/components/TagLink'
import TagSearch from '@/components/TagSearch'
import SectionTitle from '@/components/common/SectionTitle'
@@ -42,6 +43,12 @@ const TagSidebar: FC<Props> = ({ posts, onClick }) => {
const query = new URLSearchParams (location.search)
const tagsQuery = query.get ('tags') ?? ''
const anyFlg = query.get ('match') === 'any'
const countsQuery = new URLSearchParams ({ tags: tagsQuery })
for (const [key, value] of query)
{
if (key === 'match')
countsQuery.append (key, value)
}
useEffect (() => {
const tagsTmp: TagByCategory = { }
@@ -93,25 +100,37 @@ const TagSidebar: FC<Props> = ({ posts, onClick }) => {
</li>))) : [])}
</ul>
<SectionTitle>{label ('labels.relations')}</SectionTitle>
<a href="#"
onClick={ev => {
ev.preventDefault ()
void ((async () => {
try
{
const data = await apiGet<Post> ('/posts/random',
{ locale, params: { tags: tagsQuery.split (' ').filter (e => e !== '').join (' '),
match: (anyFlg ? 'any' : 'all') } })
navigate (`/posts/${ data.id }`)
}
catch
{
;
}
}) ())
}}>
{label ('actions.random')}
</a>
<ul>
<li>
<a
href="#"
onClick={ev => {
ev.preventDefault ()
void ((async () => {
try
{
const data = await apiGet<Post> (
'/posts/random',
{ locale,
params: { tags: tagsQuery.split (' ').filter (e => e !== '').join (' '),
match: (anyFlg ? 'any' : 'all') } })
navigate (`/posts/${ data.id }`)
}
catch
{
;
}
}) ())
}}>
{label ('actions.random')}
</a>
</li>
<li>
<PrefetchLink to={`/counts/posts?${ countsQuery }`}>
{label ('actions.count')}
</PrefetchLink>
</li>
</ul>
</>)
return (
+5
ファイルの表示
@@ -120,6 +120,11 @@ export const menuOutline = (
{ ...message ('entertainment'), visible: false, subMenu: [
{ ...message ('theatre'), to: '/theatres/1' },
{ ...message ('gekanator'), to: '/gekanator' }] },
{ ...message ('counts'),
name: `${ message ('counts').name } (β)`, visible: false, subMenu: [
{ ...message ('posts'), to: '/counts/posts' },
{ ...message ('tags'), to: '/counts/tags' },
{ ...message ('externalTags'), to: '/counts/nico-tags' }] },
{ ...message ('users'), to: '/users/settings', visible: false, subMenu: [
{ ...message ('list'), to: '/users', visible: false },
{ ...message ('currentUser'), to: `/users/${ user?.id }`, visible: false },
+6 -3
ファイルの表示
@@ -2,7 +2,7 @@ import { DndContext, DragOverlay, KeyboardSensor, MouseSensor, TouchSensor,
closestCenter, useSensor, useSensors } from '@dnd-kit/core'
import { SortableContext, arrayMove, horizontalListSortingStrategy,
sortableKeyboardCoordinates } from '@dnd-kit/sortable'
import { useRef, useState } from 'react'
import { useEffect, useId, useRef, useState } from 'react'
import { createPortal } from 'react-dom'
import { CSS } from '@dnd-kit/utilities'
@@ -15,6 +15,9 @@ const SortableColumns = ({ ids, fixed = [], name, onChange, children }: {
ids: string[]; fixed?: string[]; name: (id: string) => { text: string; language: string }
onChange: (ids: string[]) => void; children: (ids: string[]) => ReactNode },
) => {
const [mounted, setMounted] = useState (false)
const scope = useId ()
useEffect (() => setMounted (true), [])
const [active, setActive] = useState<string | null> (null)
const drag = useRef<{ id: string; ids: string[] } | null> (null)
const motion = useClientMotion ()
@@ -33,7 +36,7 @@ const SortableColumns = ({ ids, fixed = [], name, onChange, children }: {
drag.current = null
setActive (null)
}
return <DndContext sensors={sensors} collisionDetection={closestCenter}
return <DndContext id={scope} sensors={sensors} collisionDetection={closestCenter}
accessibility={{ announcements: {
onDragStart: event => name (String (event.active.id)).text,
onDragOver: event => event.over == null ? undefined : name (String (event.over.id)).text,
@@ -56,7 +59,7 @@ const SortableColumns = ({ ids, fixed = [], name, onChange, children }: {
<SortableContext items={ids} strategy={horizontalListSortingStrategy}>
{children (ids)}
</SortableContext>
{typeof document !== 'undefined' && createPortal (<DragOverlay adjustScale={false}
{mounted && typeof document !== 'undefined' && createPortal (<DragOverlay adjustScale={false}
style={{ width: 'max-content', height: 'auto', maxWidth: 'calc(100vw - 2rem)' }}
dropAnimation={motion.reduced ? null : {
duration: motion.duration ('drop'), easing: motion.easing,
+79 -72
ファイルの表示
@@ -3,13 +3,12 @@ import { DndContext, DragOverlay, KeyboardSensor, MouseSensor, TouchSensor,
from '@dnd-kit/core'
import { SortableContext, arrayMove, rectSortingStrategy,
sortableKeyboardCoordinates } from '@dnd-kit/sortable'
import { useId, useRef, useState } from 'react'
import { useEffect, useId, useRef, useState } from 'react'
import { createPortal } from 'react-dom'
import { CSS } from '@dnd-kit/utilities'
import SortableItem from '@/components/common/SortableItem'
import { useClientMotion } from '@/lib/useClientMotion'
import { cn } from '@/lib/utils'
import type { ReactNode } from 'react'
@@ -19,8 +18,10 @@ type Name = { text: string; language: string }
const sameGroups = (left: Groups, right: Groups): boolean =>
Object.keys (left).length === Object.keys (right).length
&& Object.keys (left).every (key => left[key].length === right[key]?.length
&& left[key].every ((id, index) => id === right[key][index]))
&& (
Object.keys (left).every (key => (
left[key].length === right[key]?.length
&& left[key].every ((id, index) => id === right[key][index]))))
const Group = ({ id, children }: { id: string; children: ReactNode }) => {
@@ -31,12 +32,14 @@ const Group = ({ id, children }: { id: string; children: ReactNode }) => {
const SortableList = (
{ groups, name, handleName, renderItem, renderGroup, onChange, fixed = [] }: {
groups: Groups; name: (id: string) => Name
handleName?: (id: string) => Name
renderItem: (id: string, group: string) => ReactNode
renderGroup: (id: string, content: ReactNode, over: boolean) => ReactNode
onChange: (groups: Groups) => void; fixed?: string[] },
groups: Groups; name: (id: string) => Name
handleName?: (id: string) => Name
renderItem: (id: string, group: string) => ReactNode
renderGroup: (id: string, content: ReactNode, over: boolean) => ReactNode
onChange: (groups: Groups) => void; fixed?: string[] },
) => {
const [mounted, setMounted] = useState (false)
useEffect (() => setMounted (true), [])
const drag = useRef<{ id: string; groups: Groups } | null> (null)
const scope = useId ()
const [active, setActive] = useState<string | null> (null)
@@ -87,69 +90,73 @@ const SortableList = (
setActive (null)
setOverGroup (null)
}
return <DndContext sensors={sensors} collisionDetection={args => {
const hits = pointerWithin (args)
const group = hits.find (hit => String (hit.id).startsWith ('group:'))
const groupName = group?.id.toString ().slice (6)
const item = groupName != null && hits.some (hit =>
owner (String (hit.id), groups) === groupName)
return group != null && !item ? [group] : closestCenter (args)
}}
accessibility={{ announcements: {
onDragStart: event => accessibleName (String (event.active.id)).text,
onDragOver: event => event.over == null ? undefined : (
event.over.id.toString ().startsWith ('group:') ? undefined
: accessibleName (String (event.over.id)).text),
onDragMove: () => undefined, onDragEnd: () => undefined,
onDragCancel: () => undefined },
screenReaderInstructions: {
draggable: active == null ? '' : accessibleName (active).text } }}
onDragStart={event => {
const id = String (event.active.id)
drag.current = { id, groups }
setActive (id)
setOverGroup (owner (id, groups) ?? null)
}} onDragOver={event => {
const id = event.over?.id.toString ()
const group = id == null ? null
: (id.startsWith ('group:') ? id.slice (6) : owner (id, groups) ?? null)
setOverGroup (previous => previous === group ? previous : group)
}} onDragCancel={reset} onDragEnd={event => {
const session = drag.current
reset ()
if (session == null || event.over == null || session.id !== String (event.active.id)
|| !sameGroups (session.groups, groups)) return
const next = relocate (groups, session.id, String (event.over.id))
if (!sameGroups (groups, next)) onChange (next)
}}>
{/* Keep indexes and droppable geometry stable until the drop is committed. */}
{Object.entries (groups).map (([group, ids]) => <Group key={group} id={group}>
{renderGroup (group, <SortableContext items={ids} strategy={rectSortingStrategy}>
<div className="flex min-h-10 flex-wrap items-start gap-1 rounded p-1">
{ids.map (id => <SortableItem key={id} id={id}
name={accessibleName (id)} fixed={fixed.includes (id)} handleId={`${scope}-${id}`}
moveGroup={direction => moveGroup (id, direction)}>
{renderItem (id, group)}
</SortableItem>)}
</div>
</SortableContext>, overGroup === group)}
</Group>)}
{typeof document !== 'undefined' && createPortal (
<DragOverlay adjustScale={false}
style={{ width: 'max-content', height: 'auto', maxWidth: 'calc(100vw - 2rem)' }}
dropAnimation={motion.reduced ? null : {
duration: motion.duration ('drop'), easing: motion.easing,
keyframes: ({ transform }) => [
{ transform: CSS.Translate.toString (transform.initial) },
{ transform: CSS.Translate.toString (transform.final) }] }}>
{active == null ? null : <div lang={name (active).language}
className="w-max max-w-[min(20rem,calc(100vw_-_2rem))] truncate rounded
border border-blue-500 bg-white px-2 py-1 text-sm
text-slate-900 shadow-lg dark:border-blue-400 dark:bg-slate-800 dark:text-slate-100">
{name (active).text}</div>}
</DragOverlay>, document.body)}
</DndContext>
return (
<DndContext
id={scope}
sensors={sensors}
collisionDetection={args => {
const hits = pointerWithin (args)
const group = hits.find (hit => String (hit.id).startsWith ('group:'))
const groupName = group?.id.toString ().slice (6)
const item = groupName != null && hits.some (hit =>
owner (String (hit.id), groups) === groupName)
return group != null && !item ? [group] : closestCenter (args)
}}
accessibility={{ announcements: {
onDragStart: event => accessibleName (String (event.active.id)).text,
onDragOver: event => event.over == null ? undefined : (
event.over.id.toString ().startsWith ('group:') ? undefined
: accessibleName (String (event.over.id)).text),
onDragMove: () => undefined, onDragEnd: () => undefined,
onDragCancel: () => undefined },
screenReaderInstructions: {
draggable: active == null ? '' : accessibleName (active).text } }}
onDragStart={event => {
const id = String (event.active.id)
drag.current = { id, groups }
setActive (id)
setOverGroup (owner (id, groups) ?? null)
}}
onDragOver={event => {
const id = event.over?.id.toString ()
const group = id == null ? null
: (id.startsWith ('group:') ? id.slice (6) : owner (id, groups) ?? null)
setOverGroup (previous => previous === group ? previous : group)
}} onDragCancel={reset} onDragEnd={event => {
const session = drag.current
reset ()
if (session == null || event.over == null || session.id !== String (event.active.id)
|| !sameGroups (session.groups, groups)) return
const next = relocate (groups, session.id, String (event.over.id))
if (!sameGroups (groups, next)) onChange (next)
}}>
{/* Keep indexes and droppable geometry stable until the drop is committed. */}
{Object.entries (groups).map (([group, ids]) => <Group key={group} id={group}>
{renderGroup (group, <SortableContext items={ids} strategy={rectSortingStrategy}>
<div className="flex min-h-10 flex-wrap items-start gap-1 rounded p-1">
{ids.map (id => <SortableItem key={id} id={id}
name={accessibleName (id)} fixed={fixed.includes (id)} handleId={`${scope}-${id}`}
moveGroup={direction => moveGroup (id, direction)}>
{renderItem (id, group)}
</SortableItem>)}
</div>
</SortableContext>, overGroup === group)}
</Group>)}
{mounted && typeof document !== 'undefined' && createPortal (
<DragOverlay adjustScale={false}
style={{ width: 'max-content', height: 'auto', maxWidth: 'calc(100vw - 2rem)' }}
dropAnimation={motion.reduced ? null : {
duration: motion.duration ('drop'), easing: motion.easing,
keyframes: ({ transform }) => [
{ transform: CSS.Translate.toString (transform.initial) },
{ transform: CSS.Translate.toString (transform.final) }] }}>
{active == null ? null : <div lang={name (active).language}
className="w-max max-w-[min(20rem,calc(100vw_-_2rem))] truncate rounded
border border-blue-500 bg-white px-2 py-1 text-sm
text-slate-900 shadow-lg dark:border-blue-400 dark:bg-slate-800 dark:text-slate-100">
{name (active).text}</div>}
</DragOverlay>, document.body)}
</DndContext>)
}
+21 -5
ファイルの表示
@@ -7,11 +7,15 @@ import { countsConditionInputs, countsEventTypes, countsFieldLabel, countsFilter
countsSetOperators, useCountsDefinition }
from '@/lib/countsDefinitions'
import useDialogue from '@/lib/dialogues/useDialogue'
import { countsConditionValue } from '@/lib/countsDisplay'
import { countsAxisLabel } from '@/lib/countsDefinitions'
import { useLocale } from '@/lib/useLocale'
import { inputClass } from '@/lib/utils'
import { useUITranslation } from '@/lib/useUITranslation'
import { readCountState, encodeCountState } from '@/lib/queryState'
import type { CountsResult } from '@/lib/counts'
import type { DialogueFormControls } from '@/lib/dialogues/useDialogue'
import type { CountsCondition, CountsDefinition } from '@/lib/countsDefinitions'
@@ -90,13 +94,15 @@ const ConditionEditor = (
}
const CountsAdditionalConditions = ({ params, update }: {
const CountsAdditionalConditions = ({ params, update, conditionLabels }: {
conditionLabels?: CountsResult['conditionLabels']
params: URLSearchParams
update: (key: string, value: string) => void },
) => {
const { label, message } = useUITranslation ('counts')
const definition = useCountsDefinition ()
const dialogue = useDialogue ()
const locale = useLocale ()
const conditions = readCountState<CountsCondition[]> (params, 'conditions', [])
const metadata = definition.filterFields ?? definition.fields
const fields = metadata.filter (field => field.operators.length > 0 && !field.legacy)
@@ -127,7 +133,7 @@ const CountsAdditionalConditions = ({ params, update }: {
{
const options: string[] = type === 'boolean' ? ['true', 'false'] : [...countsEventTypes]
const namespace = type === 'boolean' ? 'truthValues' : 'events'
return <select value={condition[key] ?? ''} disabled={condition.blank}
return <select value={condition[key] == null ? '' : String (condition[key])} disabled={condition.blank}
className={inputClass (false, 'h-10 w-36 min-w-0 max-w-full py-1')}
aria-label={aria.text}
lang={aria.language}
@@ -140,10 +146,10 @@ const CountsAdditionalConditions = ({ params, update }: {
}
if (type === 'datetime')
return <DateTimeField precision="second" className="h-10 w-48 min-w-0 max-w-full py-1"
value={condition[key] ?? ''} disabled={condition.blank}
value={condition[key] == null ? '' : String (condition[key])} disabled={condition.blank}
aria-label={aria.text} lang={aria.language}
onChange={value => patch (index, { [key]: value ?? '' })}/>
return <input value={condition[key] ?? ''} disabled={condition.blank}
return <input value={condition[key] == null ? '' : String (condition[key])} disabled={condition.blank}
type={(type === 'integer' || type === 'interval') ? 'number' : 'text'}
className={inputClass (false, 'h-10 w-36 min-w-0 max-w-full py-1')}
aria-label={aria.text} lang={aria.language}
@@ -153,7 +159,17 @@ const CountsAdditionalConditions = ({ params, update }: {
<legend className="flex items-center gap-2">{label ('additionalConditions')}
<Button type="button" variant="outline" size="sm" disabled={fields.length === 0}
onClick={() => edit ()}>{label ('add')}</Button></legend>
{conditions.map ((condition, index) => (
{conditions.map ((condition, index) => condition.operator === 'axis_eq' ? (
<div key={index} className="flex flex-wrap items-center gap-2 border-b py-2">
<span>{countsAxisLabel (condition, definition, locale).text}</span>
<span>{label ('operatorSymbols.eq')}</span>
<span>{countsConditionValue (condition.value, condition, locale,
params.get ('timezone') ?? 'UTC', conditionLabels)}</span>
<Button type="button" variant="ghost" size="sm"
aria-label={message ('remove').text}
onClick={() => write (conditions.filter ((_, position) => position !== index))}>
×</Button>
</div>) : (
<div key={index} className="flex min-w-0 flex-wrap items-center gap-x-2 gap-y-2
border-b border-slate-200 py-2 dark:border-slate-700">
<Button type="button" variant="ghost" size="sm" onClick={() => edit (index)}
+4 -3
ファイルの表示
@@ -49,9 +49,10 @@ const CountsAxes = ({ params, update }: {
commit (next)
}
return <SortableList groups={groups} name={name} onChange={commit}
renderGroup={(id, content, over) => <fieldset className="mb-2 min-w-0">
<legend>{label (id === 'available' ? 'axisItems' : id)}</legend>
<div className={cn ('rounded border dark:border-slate-700',
renderGroup={(id, content, over) => <fieldset className="mx-2 mb-2 min-w-0">
<legend className="mb-1 font-semibold">
{label (id === 'available' ? 'axisItems' : id)}</legend>
<div className={cn ('rounded border px-1 dark:border-slate-700',
over && 'ring-1 ring-blue-500 dark:ring-blue-400')}>{content}</div>
</fieldset>} renderItem={(id, group) => {
const axis = axes.find (axis => axis.field === id)
+128
ファイルの表示
@@ -0,0 +1,128 @@
import { fireEvent, screen } from '@testing-library/react'
import { useState } from 'react'
import { describe, expect, it, vi } from 'vitest'
import CountsControls from '@/components/counts/CountsControls'
import CountsFieldTree from '@/components/counts/CountsFieldTree'
import { CountsDefinitionContext } from '@/lib/countsDefinitions'
import { normaliseCountsParams } from '@/lib/counts'
import { readCountState, setCountState, encodeCountState } from '@/lib/queryState'
import { postDefinition } from '@/test/fixtures/counts'
import { renderWithProviders } from '@/test/render'
vi.mock ('@/lib/api', async original => ({
...await original<typeof import ('@/lib/api')> (), apiGet: vi.fn ().mockResolvedValue ([]) }))
const appliedParams = (extra: Record<string, string> = {}) => normaliseCountsParams ('posts',
new URLSearchParams ({ mode: 'details', timezone: 'UTC', ...extra }), postDefinition)
const renderControls = (params: URLSearchParams, labelsParams = params.toString ()) => {
const execute = vi.fn ()
const result = renderWithProviders (
<CountsDefinitionContext.Provider value={postDefinition}>
<CountsControls resource="posts" applied={params} open setOpen={vi.fn ()}
execute={execute} reorder={vi.fn ()} displayJSON={vi.fn ()}
displayRowNumbers={vi.fn ()} displayThumbnail={vi.fn ()}
conditionLabels={{ tags: { 'internal:7': 'Resolved tag' } }}
conditionLabelsParams={labelsParams}/>
</CountsDefinitionContext.Provider>, { route: '/ja/counts/posts' })
return { ...result, execute }
}
const submit = () => fireEvent.submit (
screen.getByRole ('button', { name: '集計実行' }).closest ('form')!)
describe ('Counts filter draft', () => {
it ('uses labels only for the executed condition set', () => {
const params = appliedParams ()
setCountState (params, 'conditions', encodeCountState ('conditions', [
{ field: 'tags', operator: 'axis_eq', value: 'internal:7' }]))
const { execute } = renderControls (params)
expect (screen.getByText ('Resolved tag')).toBeInTheDocument ()
fireEvent.change (screen.getByRole ('textbox', { name: 'Id.' }), { target: { value: '9' } })
expect (screen.queryByText ('Resolved tag')).not.toBeInTheDocument ()
expect (screen.getByText ('…')).toBeInTheDocument ()
expect (screen.queryByText ('internal:7')).not.toBeInTheDocument ()
submit ()
expect (readCountState (execute.mock.lastCall![0], 'conditions', []))
.toEqual ([{ field: 'tags', operator: 'axis_eq', value: 'internal:7' }])
})
it ('removes a visible drill condition from the submitted URL state', () => {
const params = appliedParams ()
setCountState (params, 'conditions', encodeCountState ('conditions', [
{ field: 'created_at', grain: 'month', operator: 'axis_eq', value: '2026-10' }]))
const { execute } = renderControls (params)
expect (screen.getByText (/令和8年10月/)).toBeInTheDocument ()
fireEvent.click (screen.getByRole ('button', { name: '削除' }))
submit ()
expect (readCountState (execute.mock.lastCall![0], 'conditions', [])).toEqual ([])
})
it ('opens the existing add dialogue without inserting a condition immediately', () => {
const { execute } = renderControls (appliedParams ())
fireEvent.click (screen.getByRole ('button', { name: '追加' }))
expect (screen.getByRole ('dialog')).toBeInTheDocument ()
expect (execute).not.toHaveBeenCalled ()
expect (document.querySelectorAll ('[role="dialog"] select.h-11').length).toBeGreaterThan (0)
expect (screen.queryByText ('pattern.input')).not.toBeInTheDocument ()
})
it ('submits only the edited tag names', () => {
const { execute } = renderControls (appliedParams ({ tags: 'alpha beta' }))
const tags = screen.getByDisplayValue ('alpha beta')
fireEvent.change (tags, { target: { value: 'not:beta gamma alpha' } })
submit ()
expect (execute.mock.lastCall![0].get ('tags')).toBe ('not:beta gamma alpha')
fireEvent.change (tags, { target: { value: 'gamma' } })
submit ()
expect (execute.mock.lastCall![0].get ('tags')).toBe ('gamma')
})
})
const FieldTree = () => {
const [params, setParams] = useState (() => appliedParams ({ columns: 'tags',
children: 'tags.name_ja' }))
return <CountsDefinitionContext.Provider value={postDefinition}>
<CountsFieldTree resource="posts" params={params} update={(key, value) => setParams (previous => {
const next = new URLSearchParams (previous)
next.set (key, value)
return next
})}/>
<output data-testid="fields">{params.toString ()}</output>
</CountsDefinitionContext.Provider>
}
it ('derives mixed parent selection and permits selecting all children again after clearing', () => {
renderWithProviders (<FieldTree/>, { route: '/ja/counts/posts' })
const parent = screen.getByRole ('checkbox', { name: 'タグ' })
expect (parent).toHaveAttribute ('aria-checked', 'mixed')
expect (parent).toBeEnabled ()
fireEvent.click (parent)
expect (parent).toBeChecked ()
fireEvent.click (parent)
expect (parent).not.toBeChecked ()
expect (parent).toBeEnabled ()
fireEvent.click (parent)
expect (parent).toBeChecked ()
const params = new URLSearchParams (screen.getByTestId ('fields').textContent ?? '')
expect (params.get ('children')?.split (' ')).toEqual ([
'tags.name_ja', 'tags.name_ain', 'tags.aliases_ja', 'tags.aliases_ain'])
expect (params.get ('columns')).toBe ('tags')
})
it ('offers row numbers only in details and keeps limit in display settings', () => {
const view = renderControls (appliedParams ({ mode: 'summary', row_numbers: '1' }))
expect (screen.queryByRole ('checkbox', { name: '行番号' })).not.toBeInTheDocument ()
const limit = screen.getByRole ('combobox', { name: '表示件数' })
expect (Array.from (limit.querySelectorAll ('option')).map (option => option.value))
.toEqual (['20', '50', '100'])
expect (limit).toHaveValue ('100')
expect (limit.querySelector ('span')).toBeNull ()
view.unmount ()
renderControls (appliedParams ({ row_numbers: '1' }))
expect (screen.getByRole ('checkbox', { name: '行番号' })).toBeChecked ()
})
+49 -9
ファイルの表示
@@ -9,7 +9,7 @@ import FormField from '@/components/common/FormField'
import HelpPopover from '@/components/common/HelpPopover'
import TagInput from '@/components/common/TagInput'
import { Button } from '@/components/ui/button'
import { countsUseLatest, fieldsForCounts, normaliseCountsParams, readCountSort,
import { countsRequestParams, countsUseLatest, fieldsForCounts, normaliseCountsParams, readCountSort,
selectedCountChildren, selectedCountColumns } from '@/lib/counts'
import { countsNestedIsScalar } from '@/lib/countsDisplay'
import { useCountsDefinition } from '@/lib/countsDefinitions'
@@ -21,13 +21,16 @@ import { readCountState, setCountState } from '@/lib/queryState'
import type { ReactNode } from 'react'
import type { CountsResource } from '@/lib/counts'
import type { CountsResource, CountsResult } from '@/lib/counts'
const CountsControls = (
{ resource, applied, open, setOpen, execute, displayJSON, displayRowNumbers,
displayThumbnail, reorder, actions, pending = false }: {
displayThumbnail, reorder, actions, conditionLabels, conditionLabelsParams,
pending = false }: {
resource: CountsResource; applied: URLSearchParams; open: boolean
conditionLabels?: CountsResult['conditionLabels']
conditionLabelsParams?: string
pending?: boolean
setOpen: (open: boolean) => void
execute: (params: URLSearchParams) => void
@@ -41,6 +44,8 @@ const CountsControls = (
const id = useId ()
const motion = useClientMotion ()
const [draft, setDraft] = useState (() => new URLSearchParams (applied))
const labelParams = conditionLabelsParams == null
? null : new URLSearchParams (conditionLabelsParams)
const thumbnail = selectedCountColumns (resource, applied, definition).includes ('thumbnail')
const previousApplied = useRef ({ resource, params: new URLSearchParams (applied) })
const appliedState = applied.toString ()
@@ -159,16 +164,16 @@ const CountsControls = (
className={inputClass (false, 'w-20 py-1')}
onChange={event => update ('limit', event.target.value)}>
{[20, 50, 100].map (count => <option key={count} value={count}>
{label ('shared:labels.resultsCount', { count })}</option>)}
{message ('shared:labels.resultsCount', { count }).text}</option>)}
</select>
</label>
<label className="inline-flex items-center gap-1 text-sm">
{details && <label className="inline-flex items-center gap-1 text-sm">
<input type="checkbox" checked={draft.get ('row_numbers') === '1'}
onChange={event => {
update ('row_numbers', event.target.checked ? '1' : '0')
displayRowNumbers (event.target.checked)
}}/>{label ('rowNumbers')}
</label>
</label>}
{details && resource === 'posts' && <label
className="inline-flex items-center gap-1 text-sm">
<input type="checkbox" checked={thumbnail}
@@ -226,10 +231,45 @@ const CountsControls = (
</FormField>)}
</div>
{resource === 'posts' && <FormField label={label ('columns.tags')}>
{() => <TagInput includeNico includeUnused value={draft.get ('tags') ?? ''}
setValue={value => update ('tags', value)}/>}
{() => <div className="flex flex-wrap items-center gap-2">
<TagInput includeNico includeUnused value={draft.get ('tags') ?? ''}
setValue={value => update ('tags', value)}/>
<select value={(draft.get ('match') === 'any'
|| draft.get ('tags')?.split (/\s+/).some (token =>
token.toLowerCase () === 'type:or')) ? 'any' : 'all'}
className={inputClass (false, 'h-10 w-auto')}
aria-label={message ('columns.tags').text}
onChange={event => {
const match = event.target.value
setDraft (previous => {
const next = new URLSearchParams (previous)
next.set ('match', match)
next.set ('tags', (next.get ('tags') ?? '').split (/\s+/)
.filter (token => token.toLowerCase () !== 'type:or').join (' '))
next.delete ('page')
return next
})
}}>
<option value="all">AND</option><option value="any">OR</option>
</select>
{draft.get ('tags') && <>
<Button type="button" variant="ghost" size="sm"
aria-label={message ('remove').text} onClick={() => setDraft (previous => {
const next = new URLSearchParams (previous)
for (const key of [...next.keys ()])
if (['tags', 'tag', 'kind', 'match', 'page'].includes (key)) next.delete (key)
return next
})}>×</Button>
</>}
</div>}
</FormField>}
<CountsAdditionalConditions params={draft} update={update}/>
<CountsAdditionalConditions params={draft} update={update}
conditionLabels={(labelParams != null
&& draft.get ('current') === labelParams.get ('current')
&& draft.get ('at') === labelParams.get ('at')
&& countsRequestParams (draft).toString ()
=== countsRequestParams (labelParams).toString ())
? conditionLabels : undefined}/>
</fieldset>
</div>
</div>
+198
ファイルの表示
@@ -0,0 +1,198 @@
import { act, cleanup, screen } from '@testing-library/react'
import { StrictMode } from 'react'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import CountsExportProgress from '@/components/counts/CountsExportProgress'
import { renderWithProviders } from '@/test/render'
const api = vi.hoisted (() => ({
apiPost: vi.fn (), apiGet: vi.fn (), apiDelete: vi.fn (), startAPIDownload: vi.fn () }))
vi.mock ('@/lib/api', async original => ({
...await original<typeof import ('@/lib/api')> (), ...api }))
const ready = { status: 'ready', processed: 5, total: 5, token: 'file' }
const receipt = { downloadAttempt: 'attempt', downloadStatus: 'pending', token: 'file' }
const controls = () => ({ close: vi.fn (), setActions: vi.fn (), confirm: vi.fn () })
const flush = async (milliseconds = 0) => act (async () => {
await vi.advanceTimersByTimeAsync (milliseconds)
})
beforeEach (() => {
vi.useFakeTimers ()
Object.values (api).forEach (mock => mock.mockReset ())
api.apiPost.mockImplementation (async (path: string) =>
path.endsWith ('/download') ? receipt : { job: 'job' })
api.apiGet.mockResolvedValue (ready)
api.apiDelete.mockResolvedValue ({ status: 'cancelled', processed: 5, total: 5 })
api.startAPIDownload.mockReturnValue (vi.fn ())
})
afterEach (() => { cleanup (); vi.useRealTimers () })
const renderProgress = (format = 'csv') => {
const actions = controls ()
const handle = { cancel: async () => false }
const element = (next = actions) => <StrictMode>
<CountsExportProgress payload={{ resource: 'posts', format }} locale="ja"
handle={handle} controls={next}/>
</StrictMode>
return { ...renderWithProviders (element ()), actions, handle, element }
}
describe ('Counts native download acknowledgement', () => {
it.each (['csv', 'json']) ('closes %s after the matching GET acknowledgement', async format => {
api.apiGet.mockResolvedValueOnce (ready).mockResolvedValue ({ ...ready,
downloadAttempt: 'attempt', downloadStatus: 'started' })
const view = renderProgress (format)
await flush ()
expect (api.apiPost).toHaveBeenCalledTimes (2)
expect (api.startAPIDownload).toHaveBeenCalledTimes (1)
expect (view.actions.close).not.toHaveBeenCalled ()
const query = api.startAPIDownload.mock.calls[0][3] as URLSearchParams
expect (Object.fromEntries (query)).toEqual ({ job: 'job', attempt: 'attempt' })
const next = controls ()
view.rerender (view.element (next))
await flush (800)
expect (next.close).toHaveBeenCalledTimes (1)
expect (api.startAPIDownload).toHaveBeenCalledTimes (1)
})
it ('times out independently of pending status requests', async () => {
api.apiGet.mockResolvedValueOnce (ready).mockImplementation (() => new Promise (() => {}))
const view = renderProgress ()
await flush ()
await flush (30_000)
expect (screen.getByRole ('alert')).toHaveTextContent ('タイムアウト')
expect (view.actions.close).not.toHaveBeenCalled ()
expect (api.startAPIDownload.mock.results[0].value).toHaveBeenCalled ()
})
it ('retains the modal after acceptance failure without repeated POSTs', async () => {
api.apiPost.mockResolvedValueOnce ({ job: 'job' })
.mockRejectedValueOnce (new Error ('offline'))
const view = renderProgress ()
await flush ()
expect (screen.getByRole ('alert')).toHaveTextContent ('開始できません')
await flush (40_000)
expect (api.apiPost).toHaveBeenCalledTimes (2)
expect (api.startAPIDownload).not.toHaveBeenCalled ()
expect (view.actions.close).not.toHaveBeenCalled ()
})
it ('retries a temporary status transport failure without a second native GET', async () => {
api.apiGet.mockResolvedValueOnce (ready).mockRejectedValueOnce (new Error ('offline'))
.mockResolvedValue ({ ...ready, downloadAttempt: 'attempt', downloadStatus: 'started' })
const view = renderProgress ()
await flush ()
await flush (2800)
expect (view.actions.close).toHaveBeenCalledTimes (1)
expect (api.startAPIDownload).toHaveBeenCalledTimes (1)
})
it ('cancels an acceptance in flight and never starts the resulting receipt', async () => {
let resolve!: (value: typeof receipt) => void
api.apiPost.mockResolvedValueOnce ({ job: 'job' }).mockImplementationOnce (
() => new Promise (complete => { resolve = complete }))
const view = renderProgress ()
await flush ()
await act (async () => { expect (await view.handle.cancel ()).toBe (true) })
await act (async () => { resolve (receipt) })
await flush (1000)
expect (api.startAPIDownload).not.toHaveBeenCalled ()
expect (view.actions.close).not.toHaveBeenCalled ()
})
it ('releases an unmounted iframe and ignores late acknowledgements', async () => {
const view = renderProgress ()
await flush ()
const release = api.startAPIDownload.mock.results[0].value
view.unmount ()
await flush ()
expect (release).toHaveBeenCalledTimes (1)
await flush (40_000)
expect (view.actions.close).not.toHaveBeenCalled ()
expect (api.startAPIDownload).toHaveBeenCalledTimes (1)
})
it ('keeps authentication expiry and iframe errors visible', async () => {
const view = renderProgress ()
await flush ()
act (() => { api.startAPIDownload.mock.calls[0][2] () })
expect (screen.getByRole ('alert')).toHaveTextContent ('開始できません')
await flush (40_000)
expect (view.actions.close).not.toHaveBeenCalled ()
})
it ('shows the worker-specific reason instead of a download-success message', async () => {
api.apiPost.mockRejectedValueOnce ({ isAxiosError: true,
response: { status: 503, data: { error: 'export_worker_unavailable' } } })
const view = renderProgress ()
await flush ()
expect (screen.getByRole ('alert')).toHaveTextContent ('ワーカーが稼働していません')
expect (view.actions.close).not.toHaveBeenCalled ()
expect (await view.handle.cancel ()).toBe (true)
})
it ('shows revoked authority and does not close after a successful acceptance', async () => {
api.apiGet.mockResolvedValueOnce (ready).mockRejectedValue ({ isAxiosError: true,
response: { status: 404 } })
const view = renderProgress ()
await flush ()
await flush (800)
expect (screen.getByRole ('alert')).toHaveTextContent ('認可が失われたか')
expect (view.actions.close).not.toHaveBeenCalled ()
})
it ('stops after cancellation failure and permits an explicit cancellation retry', async () => {
const view = renderProgress ()
await flush ()
api.apiDelete.mockRejectedValueOnce (new Error ('offline'))
await act (async () => { expect (await view.handle.cancel ()).toBe (false) })
const requests = api.apiGet.mock.calls.length
await flush (40_000)
expect (api.apiGet).toHaveBeenCalledTimes (requests)
expect (screen.getByRole ('alert')).toBeInTheDocument ()
await act (async () => { expect (await view.handle.cancel ()).toBe (true) })
expect (api.apiDelete).toHaveBeenCalledTimes (2)
expect (api.startAPIDownload).toHaveBeenCalledTimes (1)
})
it ('ignores a different attempt even when it reports started', async () => {
api.apiGet.mockResolvedValueOnce (ready).mockResolvedValueOnce ({ ...ready,
downloadAttempt: 'another', downloadStatus: 'started' }).mockResolvedValue ({ ...ready,
downloadAttempt: 'attempt', downloadStatus: 'started' })
const view = renderProgress ()
await flush ()
await flush (800)
expect (view.actions.close).not.toHaveBeenCalled ()
await flush (800)
expect (view.actions.close).toHaveBeenCalledTimes (1)
})
it ('keeps the created job available for cancellation retry during initial registration', async () => {
let resolve!: (value: { job: string }) => void
api.apiPost.mockImplementationOnce (() => new Promise (complete => { resolve = complete }))
api.apiDelete.mockRejectedValueOnce (new Error ('offline'))
const view = renderProgress ()
await flush ()
let pending!: Promise<boolean>
act (() => { pending = view.handle.cancel () })
await act (async () => {
resolve ({ job: 'created-during-cancel' })
expect (await pending).toBe (false)
})
await act (async () => { expect (await view.handle.cancel ()).toBe (true) })
expect (api.apiDelete.mock.calls.map (call => call[0])).toEqual ([
'/counts/export/jobs/created-during-cancel', '/counts/export/jobs/created-during-cancel'])
expect (api.startAPIDownload).not.toHaveBeenCalled ()
})
it ('stops immediately if native download navigation cannot be created', async () => {
api.startAPIDownload.mockImplementationOnce (() => { throw new Error ('navigation blocked') })
const view = renderProgress ()
await flush ()
expect (screen.getByRole ('alert')).toHaveTextContent ('開始できません')
await flush (40_000)
expect (api.apiGet).toHaveBeenCalledTimes (1)
expect (view.actions.close).not.toHaveBeenCalled ()
})
})
+181 -70
ファイルの表示
@@ -2,6 +2,7 @@ import { useEffect, useRef, useState } from 'react'
import LocalisedText from '@/components/LocalisedText'
import { apiDelete, apiGet, apiPost, isApiError, startAPIDownload } from '@/lib/api'
import { countsErrorMessage } from '@/lib/countsDisplay'
import { useClientMotion } from '@/lib/useClientMotion'
import { useUITranslation } from '@/lib/useUITranslation'
import { cn } from '@/lib/utils'
@@ -13,6 +14,7 @@ export type CountsExportHandle = { cancel: () => Promise<boolean> }
type ExportStatus = {
status: 'starting' | 'queued' | 'running' | 'cancelling' | 'cancelled' | 'ready' | 'failed'
processed: number; total: number | null; token?: string
downloadAttempt?: string; downloadStatus?: string; error?: string
phase?: 'queued' | 'aggregating' | 'writing'; heartbeatAt?: number; updatedAt?: number }
@@ -25,126 +27,228 @@ const CountsExportProgress = (
const motion = useClientMotion ()
const [state, setState] = useState<ExportStatus> ({
status: 'starting', processed: 0, total: null })
const [cancelError, setCancelError] = useState (false)
const [cancelError, setCancelError] = useState<unknown> (null)
const [failure, setFailure] = useState<unknown> (null)
const [downloadError, setDownloadError] = useState<string | null> (null)
// One mounted dialogue owns one immutable job, even if its parent rerenders.
const initial = useRef ({ payload, locale })
const actions = useRef (controls)
actions.current = controls
const terminal = useRef (false)
const finished = useRef (false)
const cancelAction = useRef<() => Promise<boolean>> (async () => false)
const cancelled = useRef (false)
const generation = useRef (0)
const job = useRef<string | null> (null)
const creation = useRef<Promise<{ job: string }> | null> (null)
const downloaded = useRef (false)
const pollRequest = useRef<AbortController | null> (null)
const registration = useRef<Promise<ExportStatus> | null> (null)
const attempt = useRef<string | null> (null)
const downloadDeadline = useRef (0)
const releaseDownload = useRef<(() => void) | null> (null)
const cancellation = useRef<Promise<boolean> | null> (null)
useEffect (() => { handle.cancel = () => cancelAction.current () }, [handle])
useEffect (() => {
handle.cancel = async () => {
if (terminal.current) return true
const epoch = ++generation.current
let active = true
let lastResponse = Date.now ()
let timer: ReturnType<typeof setTimeout> | null = null
let acknowledgementTimer: ReturnType<typeof setTimeout> | null = null
let request: AbortController | null = null
const current = () => active && generation.current === epoch
const stop = (reason: string) => {
if (!current () || terminal.current) return
terminal.current = true
if (timer != null) clearTimeout (timer)
if (acknowledgementTimer != null) clearTimeout (acknowledgementTimer)
request?.abort ()
releaseDownload.current?.()
releaseDownload.current = null
setDownloadError (reason)
}
const cancel = (): Promise<boolean> => {
if (finished.current || (terminal.current && job.current == null))
return Promise.resolve (true)
cancelled.current = true
if (job.current != null)
{
terminal.current = true
if (timer != null) clearTimeout (timer)
if (acknowledgementTimer != null) clearTimeout (acknowledgementTimer)
request?.abort ()
releaseDownload.current?.()
releaseDownload.current = null
cancellation.current ??= (async () => {
const controller = new AbortController ()
const deadline = setTimeout (() => controller.abort (), 15_000)
try
{
setCancelError (false)
const created = await creation.current
if (created == null) return false
job.current = created.job
const result = await apiDelete<ExportStatus> (
`/counts/export/jobs/${encodeURIComponent (job.current)}`, {
locale, signal: controller.signal })
setState (result)
`/counts/export/jobs/${encodeURIComponent (created.job)}`, {
locale: initial.current.locale, signal: controller.signal })
if (current ())
{
setState (result)
setCancelError (null)
finished.current = ['cancelled', 'failed'].includes (result.status)
terminal.current = finished.current
if (!terminal.current)
{
if (timer != null) clearTimeout (timer)
timer = setTimeout (() => poll (created.job), 800)
}
}
return ['cancelled', 'failed'].includes (result.status)
}
catch
catch (error)
{
setCancelError (true)
if (isApiError (error) && error.response?.status === 404) return true
if (current ()) setCancelError (error)
return false
}
finally
{
clearTimeout (deadline)
cancellation.current = null
}
}
return false
}) ()
return cancellation.current
}
cancelAction.current = cancel
const armDeadline = () => {
if (downloadDeadline.current === 0) return
acknowledgementTimer = setTimeout (() => stop ('timeout'),
Math.max (0, downloadDeadline.current - Date.now ()))
}
}, [handle, locale])
useEffect (() => {
let active = true
let lastResponse = Date.now ()
let timer: ReturnType<typeof setTimeout> | undefined
const poll = async (token: string) => {
if (!current () || terminal.current) return
const controller = new AbortController ()
pollRequest.current = controller
request = controller
const deadline = setTimeout (() => controller.abort (), 15_000)
try
{
const response = await apiGet<ExportStatus> (
const result = await apiGet<ExportStatus> (
`/counts/export/jobs/${encodeURIComponent (token)}`, {
locale, signal: controller.signal })
if (!active) return
const result = (cancelled.current && response.status === 'ready')
? await apiDelete<ExportStatus> (
`/counts/export/jobs/${encodeURIComponent (token)}`, {
locale, signal: controller.signal }) : response
if (!active) return
locale: initial.current.locale, signal: controller.signal })
if (!current () || terminal.current) return
lastResponse = Date.now ()
setState (result)
terminal.current = ['ready', 'failed', 'cancelled'].includes (result.status)
if (result.status === 'ready' && result.token != null && !downloaded.current)
if (cancelled.current)
{
downloaded.current = true
startAPIDownload ('/counts/export', result.token, () => {
if (active) setState (previous => ({ ...previous, status: 'failed' }))
})
if (await cancel ()) return
}
if (!terminal.current) timer = setTimeout (() => poll (token), 800)
else if (result.status === 'ready')
{
if (result.token == null) { stop ('failed'); return }
if (attempt.current == null)
{
registration.current ??= apiPost<ExportStatus> (
`/counts/export/jobs/${encodeURIComponent (token)}/download`, {}, {
locale: initial.current.locale, signal: controller.signal })
const receipt = await registration.current
if (!current () || terminal.current || cancelled.current) return
if (receipt.downloadAttempt == null) { stop ('failed'); return }
attempt.current = receipt.downloadAttempt
downloadDeadline.current = Date.now () + 30_000
armDeadline ()
try
{
releaseDownload.current = startAPIDownload ('/counts/export', result.token,
() => stop ('failed'), new URLSearchParams ({
job: token, attempt: receipt.downloadAttempt }))
}
catch
{
stop ('failed')
return
}
}
if (result.downloadAttempt === attempt.current)
{
if (result.downloadStatus === 'started')
{
terminal.current = true
finished.current = true
if (acknowledgementTimer != null) clearTimeout (acknowledgementTimer)
// Rack yielded a chunk; browser receipt and saving are not confirmed.
releaseDownload.current = null
actions.current.close ()
return
}
if (['failed', 'unavailable', 'timeout'].includes (result.downloadStatus ?? ''))
{ stop (result.downloadStatus!); return }
}
}
else if (['failed', 'cancelled'].includes (result.status))
{
finished.current = true
terminal.current = true
return
}
if (timer != null) clearTimeout (timer)
timer = setTimeout (() => poll (token), 800)
}
catch (error)
{
if (active)
{
if ((isApiError (error) && error.response?.status === 404)
|| Date.now () - lastResponse > 30_000)
{
terminal.current = true
setState (previous => ({ ...previous, status: 'failed' }))
return
}
setCancelError (true)
timer = setTimeout (() => poll (token), 2000)
}
if (!current () || terminal.current) return
const status = isApiError (error) ? error.response?.status : null
if ([400, 401, 403, 422].includes (status ?? 0)) setFailure (error)
if (registration.current != null && attempt.current == null)
{ stop ('failed'); return }
if (status != null && status >= 400 && status < 500)
{ stop ('unavailable'); return }
if (Date.now () - lastResponse >= 30_000)
{ stop ('timeout'); return }
timer = setTimeout (() => poll (token), 2000)
}
finally
{
clearTimeout (deadline)
if (pollRequest.current === controller) pollRequest.current = null
if (request === controller) request = null
}
}
if (creation.current == null)
{
const controller = new AbortController ()
const deadline = setTimeout (() => controller.abort (), 15_000)
creation.current = apiPost<{ job: string }> ('/counts/export/jobs', payload, {
locale, signal: controller.signal }).finally (() => clearTimeout (deadline))
creation.current = apiPost<{ job: string }> (
'/counts/export/jobs', initial.current.payload, {
locale: initial.current.locale, signal: controller.signal })
.finally (() => clearTimeout (deadline))
}
creation.current.then (async result => {
if (!active) return
armDeadline ()
void creation.current.then (result => {
if (!current () || terminal.current) return
job.current = result.job
if (cancelled.current) await handle.cancel ()
await poll (result.job)
}).catch (() => {
if (active)
{
terminal.current = true
setState (previous => ({ ...previous, status: 'failed' }))
}
void poll (result.job)
}).catch (error => {
if (current ()) setFailure (error)
stop ('failed')
})
return () => {
active = false
if (timer != null) clearTimeout (timer)
pollRequest.current?.abort ()
if (acknowledgementTimer != null) clearTimeout (acknowledgementTimer)
request?.abort ()
// StrictMode immediately installs another effect; real unmount releases the frame.
queueMicrotask (() => {
if (generation.current !== epoch) return
releaseDownload.current?.()
releaseDownload.current = null
})
}
}, [handle, locale, payload])
useEffect (() => {
controls.setActions ([])
}, [controls])
}, [])
useEffect (() => { controls.setActions ([]) }, [controls])
const percent = state.total == null ? null : (state.total === 0 ? 100
: Math.min (100, Math.floor (state.processed / state.total * 100)))
const busy = !['ready', 'failed', 'cancelled'].includes (state.status)
const phase = message (`exportProgress.${state.status === 'running'
? state.phase ?? 'running' : state.status}`)
const busy = downloadError == null && cancelError == null
&& !['failed', 'cancelled'].includes (state.status)
const phaseKey = (downloadError != null || cancelError != null) ? 'failed'
: (state.status === 'running' ? state.phase ?? 'running'
: (state.status === 'ready' ? 'writing' : state.status))
const phase = message (`exportProgress.${phaseKey}`)
const failureMessage = countsErrorMessage (failure ?? state, locale)
const cancellationMessage = countsErrorMessage (cancelError, locale)
return <div className="space-y-3 py-2" aria-busy={busy}>
<p>{label ('exportProgress.format', {
format: message (`exportFormats.${payload.format}`).text })}</p>
@@ -161,7 +265,14 @@ const CountsExportProgress = (
? label ('exportProgress.processed', { count: state.processed })
: label ('exportProgress.measured', {
processed: state.processed, total: state.total ?? 0, percent })}</p>
{cancelError && <p role="alert">{label ('common:fetchError')}</p>}
{(downloadError != null || state.status === 'failed') && <p role="alert"
lang={failureMessage.language}>
{failure != null || state.status === 'failed'
? failureMessage.text
: label (`downloadErrors.${downloadError}`)}
{downloadError != null && <><br/>{label ('errors.stopped')}</>}</p>}
{cancelError != null && <p role="alert" lang={cancellationMessage.language}>
{cancellationMessage.text}</p>}
</div>
}
+2 -1
ファイルの表示
@@ -72,7 +72,8 @@ const CountsNestedTable = (
{params.get ('nested_json') === '1' ? (
<pre data-counts-scroll
aria-label={title.text} lang={title.language}
className="max-h-72 w-full overflow-auto whitespace-pre text-xs">
className="max-h-72 w-full overflow-y-auto whitespace-pre-wrap
[overflow-wrap:anywhere] font-sans text-xs">
{JSON.stringify (countsNestedExport (rows, fields, path, params, locale))}
</pre>) : (columns.length > 0 && rows.length > 0 && (
<div data-counts-scroll
+12 -6
ファイルの表示
@@ -3,6 +3,7 @@ import { useContext, useEffect, useRef } from 'react'
import SortableItem from '@/components/common/SortableItem'
import CountsResizeHandle from '@/components/counts/CountsResizeHandle'
import { Button } from '@/components/ui/button'
import { cn } from '@/lib/utils'
import { useUITranslation } from '@/lib/useUITranslation'
import { CountsTableScope, useCountsTableLayout } from '@/lib/countsTableLayout'
@@ -10,14 +11,15 @@ import type { CountSort } from '@/lib/counts'
const CountsSortHeader = (
{ field, label, sorts, change, structured = false, rowSpan, colSpan, resizable = true, dragId, totalKind }: {
{ field, label, sorts, change, structured = false, rowSpan, colSpan,
resizable = true, dragId, totalKind, disabled = false }: {
field: string
label: { text: string; language: string }
sorts: CountSort[]
change: (key: string, additive: boolean) => void
structured?: boolean
rowSpan?: number; colSpan?: number; resizable?: boolean
dragId?: string; totalKind?: string },
dragId?: string; totalKind?: string; disabled?: boolean },
) => {
const { message } = useUITranslation ('counts')
const family = useContext (CountsTableScope)
@@ -83,7 +85,7 @@ const CountsSortHeader = (
aria-sort={(sorts.length > 1 && priority >= 0) ? 'other' : state}>
<div ref={heading} className="flex min-w-0 items-center">
{shell?.handle}
<Button type="button" variant="ghost" size="sm"
<Button type="button" variant="ghost" size="sm" disabled={disabled}
className="h-auto w-full max-w-full justify-start select-none px-1 py-0 pr-3"
lang={aria.language} aria-label={aria.text} title={title.text}
onPointerDown={event => {
@@ -129,7 +131,10 @@ const CountsSortHeader = (
}
change (field, event.ctrlKey || event.metaKey)
}}>
<span data-counts-label className="truncate" lang={label.language}>{label.text}</span>
<span data-counts-label className={cn ('truncate', label.text === '********'
&& 'text-red-600 dark:text-red-400')}
title={label.text === '********' ? message ('maskedReason').text : undefined}
lang={label.language}>{label.text}</span>
<span aria-hidden data-counts-sort-indicator className="inline-block w-10 shrink-0">
{priority >= 0 && <>{direction === 'asc' ? '▲' : '▼'}
{sorts.length > 1 && priority + 1}</>}</span>
@@ -137,8 +142,9 @@ const CountsSortHeader = (
</div>
{resizable && <CountsResizeHandle field={field} label={label}/>}
</th>)
return dragId == null ? render () : <SortableItem id={dragId} name={message ('reorder', { column: label.text })}
handleId={`column-${dragId}`} moveGroup={() => { }} renderShell={render}>{null}</SortableItem>
return dragId == null ? render () : <SortableItem id={dragId}
name={message ('reorder', { column: label.text })} handleId={`column-${dragId}`}
moveGroup={() => { }} renderShell={render}>{null}</SortableItem>
}
+46
ファイルの表示
@@ -0,0 +1,46 @@
import { fireEvent, screen } from '@testing-library/react'
import { describe, expect, it, vi } from 'vitest'
import CountsSummaryResults from '@/components/counts/CountsSummaryResults'
import CountsTableLayout from '@/components/counts/CountsTableLayout'
import { CountsDefinitionContext } from '@/lib/countsDefinitions'
import { postDefinition } from '@/test/fixtures/counts'
import { renderWithProviders } from '@/test/render'
import type { CountsResult } from '@/lib/counts'
const result: CountsResult = {
count: 4, page: 1, limit: 100, groupCount: 1, outputCount: 1,
rows: [{ field: 'resource_id' }], cols: [], columnValuesEditable: false,
rowHeaders: [{ values: ['__counts_private__'], kind: 'data' },
{ values: [], kind: 'grand_total' }],
columnHeaders: [{ values: [], kind: 'grand_total' }],
cells: [{ values: ['__counts_private__'], count: 4 }],
pivotTotals: [{ row: [], column: [], count: 4 }], axisLabels: {} }
describe ('Counts summary privacy and row numbering', () => {
it ('copies masked labels and omits private links and legacy numbering', () => {
const { container } = renderWithProviders (
<CountsDefinitionContext.Provider value={postDefinition}>
<CountsTableLayout><CountsSummaryResults resource="posts" data={result}
params={new URLSearchParams ('mode=summary&row_numbers=1&rows=resource_id&cols=')}
pending={false} update={vi.fn ()}/></CountsTableLayout>
</CountsDefinitionContext.Provider>, { route: '/ja/counts/posts' })
const label = screen.getByText ('********')
expect (label).toHaveAttribute ('title', expect.stringContaining ('集計件数には含まれます'))
expect (label).toHaveClass ('text-red-600', 'dark:text-red-400')
const row = label.closest ('tr')!
expect (row.querySelector ('a')).toBeNull ()
expect (container.innerHTML).not.toContain ('__counts_private__')
expect (screen.queryByText ('No.')).not.toBeInTheDocument ()
expect (row.querySelectorAll ('td')).toHaveLength (2)
const table = screen.getByRole ('grid')
table.focus ()
fireEvent.keyDown (table, { key: 'Home' })
const setData = vi.fn ()
fireEvent.copy (table, { clipboardData: { setData } })
const plain = setData.mock.calls.find (entry => entry[0] === 'text/plain')?.[1]
expect (plain).toContain ('********')
expect (plain).not.toContain ('__counts_private__')
})
})
+35 -29
ファイルの表示
@@ -1,17 +1,18 @@
import SortableColumns from '@/components/common/SortableColumns'
import { countsSummaryValue } from '@/lib/countsDisplay'
import PrefetchLink from '@/components/PrefetchLink'
import CountsRowNumber from '@/components/counts/CountsRowNumber'
import CountsSelectableTable from '@/components/counts/CountsSelectableTable'
import CountsSortHeader from '@/components/counts/CountsSortHeader'
import { changeCountSort, countsRequestParams, readCountSort, writeCountSort } from '@/lib/counts'
import { changeCountSort, countsMaskedValue, countsRequestParams, countsResourcePath,
readCountSort, writeCountSort }
from '@/lib/counts'
import { countsAxisLabel, useCountsDefinition }
from '@/lib/countsDefinitions'
import { cn } from '@/lib/utils'
import { useLocale } from '@/lib/useLocale'
import { useUITranslation } from '@/lib/useUITranslation'
import { queryRecordParams, setQueryRecords, readCountState, encodeCountState, setCountState } from '@/lib/queryState'
import { queryRecordParams, setQueryRecords, readCountState, encodeCountState, setCountState }
from '@/lib/queryState'
import type { CountsPivotHeader, CountsResource, CountsResult } from '@/lib/counts'
import type { CountsAxis } from '@/lib/countsDefinitions'
@@ -21,12 +22,12 @@ const CountsSummaryTable = (
{ resource, data, params, pending, update, dragIds = {} }: {
dragIds?: Record<string, string>
resource: CountsResource; data: CountsResult; params: URLSearchParams
pending: boolean; update: (key: string, value: string, extra?: Record<string, string>) => void },
pending: boolean
update: (key: string, value: string, extra?: Record<string, string>) => void },
) => {
const definition = useCountsDefinition ()
const locale = useLocale ()
const { message } = useUITranslation ('counts')
const numbered = params.get ('row_numbers') === '1'
const rows = data.rows ?? []
const cols = data.cols ?? []
const cells = data.cells ?? []
@@ -49,7 +50,8 @@ const CountsSummaryTable = (
const headingLevels = cols.length === 0 ? 1 : levels * 2
const names = (header: CountsPivotHeader, axes: CountsAxis[], offset: number) =>
header.values.map ((value, index) => display (value, axes[index], offset + index))
const columnKey = (header: CountsPivotHeader) => `${header.kind}:${JSON.stringify (header.values)}`
const columnKey = (header: CountsPivotHeader) =>
`${header.kind}:${JSON.stringify (header.values)}`
const measures = readCountState<Record<string, string | number | null>[]> (
params, 'measures', [])
const measureKey = (header: CountsPivotHeader) => {
@@ -66,13 +68,14 @@ const CountsSummaryTable = (
const next = new URLSearchParams (params)
const nextSort = changeCountSort (sorts, key, additive)
const active = new Set (nextSort.map (sort => sort.key))
const records = measures.filter (record => record.key !== key && active.has (String (record.key)))
const records = measures.filter (record =>
record.key !== key && active.has (String (record.key)))
setQueryRecords (next, 'measures', [...records, ...(active.has (key)
? header.values.map ((value, axis) => ({ key, axis, value })) : [])])
update ('sort', writeCountSort (nextSort),
queryRecordParams (next, 'measures'))
}
const keys = [...(numbered ? ['row_number'] : []),
const keys = [
...Array.from ({ length: leading }, (_, index) => `a${index}`),
...columnHeaders.map (columnKey)]
const full = new Map (cells.map (cell => [JSON.stringify (cell.values), cell.count]))
@@ -86,22 +89,21 @@ const CountsSummaryTable = (
: totals.get (JSON.stringify ([row.values, column.values])) ?? 0
const drill = (row: CountsPivotHeader, column: CountsPivotHeader) => {
const next = new URLSearchParams (params)
const existing = readCountState<Record<string, string | number | null>[]> (params, 'drill', [])
setCountState (next, 'drill', encodeCountState ('drill', [...existing,
...row.values.map ((value, index) => ({ ...rows[index], value })),
...column.values.map ((value, index) => ({ ...cols[index], value }))]))
const existing = readCountState<Record<string, string | number | null>[]> (
params, 'conditions', [])
setCountState (next, 'conditions', encodeCountState ('conditions', [...existing,
...row.values.map ((value, index) => ({ ...rows[index], operator: 'axis_eq', value })),
...column.values.map ((value, index) => ({
...cols[index], operator: 'axis_eq', value }))]))
next.set ('mode', 'details')
next.delete ('sort')
next.delete ('page')
return `/counts/${resource}?${next}`
return `${countsResourcePath (resource)}?${next}`
}
let ordinal = (data.page - 1) * data.limit
const rowLabels = rowHeaders.map (header => {
const labels = names (header, rows, 0)
// With no row axes the sole result row is not an additional rollup row.
const number = rows.length === 0 ? '1' : (header.kind === 'data' ? String (++ordinal) : '')
if (header.kind !== 'data') labels.push (aggregateLabel (header))
return [...(numbered ? [number] : []),
return [
...Array.from ({ length: leading }, (_, index) => labels[index] ?? '')]
})
const spans = rowHeaders.map ((row, index) => rows.map ((_, axis) => {
@@ -121,11 +123,10 @@ const CountsSummaryTable = (
return span
}))
rowLabels.forEach ((labels, row) => spans[row].forEach ((span, column) => {
if (span === 0) labels[column + (numbered ? 1 : 0)] = ''
if (span === 0) labels[column] = ''
}))
// This unmerged matrix is also the single source for TSV and HTML clipboard headings.
const valueHeadingRows = Array.from ({ length: levels }, (_, level) => [
...(numbered ? [level === 0 ? message ('rowNumberHeading').text : ''] : []),
...Array.from ({ length: leading }, (_, index) =>
level === 0 && rows[index] != null
? countsAxisLabel (rows[index], definition, locale).text : ''),
@@ -142,7 +143,7 @@ const CountsSummaryTable = (
return display (header.values[level], cols[level], rows.length + level)
})])
const axisHeadingRows = cols.map ((axis, level) => [
...(numbered ? [''] : []), ...Array.from ({ length: leading }, () => ''),
...Array.from ({ length: leading }, () => ''),
...columnHeaders.map ((header, index) => {
if (header.values.length < level) return ''
const previous = columnHeaders[index - 1]
@@ -155,7 +156,7 @@ const CountsSummaryTable = (
: axisHeadingRows.flatMap ((headings, level) => [headings, valueHeadingRows[level]])
if (cols.length > 0)
{
for (let index = 0; index < leading + (numbered ? 1 : 0); ++index)
for (let index = 0; index < leading; ++index)
{
headingRows[0][index] = valueHeadingRows[0][index]
headingRows[1][index] = ''
@@ -214,6 +215,7 @@ const CountsSummaryTable = (
const field = leaf ? columnKey (header) : sortKey
output.push (<CountsSortHeader key={`${level}:${field}:${index}`} field={field}
colSpan={span} resizable={leaf}
disabled={header.values.includes (countsMaskedValue)}
rowSpan={(aggregate && cols.length > 0) ? (levels - level) * 2 - 1 : 1}
totalKind={header.kind !== 'data' ? header.kind : undefined}
dragId={level === 0 && header.values.length > 0
@@ -235,7 +237,6 @@ const CountsSummaryTable = (
<thead className="sticky top-0 z-20">
{headingRows.map ((_, tier) => <tr key={tier}>
{tier === 0 && <>
{numbered && <CountsRowNumber rowSpan={headingLevels}/>}
{Array.from ({ length: leading }, (_, index) => rows[index] == null
? <th key={index} rowSpan={headingLevels}/>
: <CountsSortHeader key={rows[index].field} field={`a${index}`}
@@ -249,7 +250,7 @@ const CountsSummaryTable = (
</thead>
<tbody>{rowHeaders.map ((row, index) => <tr key={columnKey (row)}>
{rowLabels[index].map ((value, offset) => {
const axis = offset - (numbered ? 1 : 0)
const axis = offset
if (axis >= 0 && row.kind !== 'data' && axis > row.values.length) return null
const aggregate = axis >= 0 && row.kind !== 'data' && axis === row.values.length
const span = axis < 0 ? 1 : spans[index][axis] ?? 1
@@ -259,8 +260,10 @@ const CountsSummaryTable = (
data-counts-aggregate={row.kind !== 'data' ? '' : undefined}
data-counts-total={row.kind !== 'data' ? row.kind : undefined}
data-counts-row-total={row.kind !== 'data' ? row.kind : undefined}
className={cn ('overflow-hidden break-words border px-2 py-1 dark:border-slate-700',
numbered && offset === 0 && 'text-right tabular-nums')}>{value}</td>
className="overflow-hidden break-words border px-2 py-1 dark:border-slate-700">
{value === '********' ? <span
className="text-red-600 dark:text-red-400"
title={message ('maskedReason').text}>{value}</span> : value}</td>
})}
{columnHeaders.map (column => <td key={columnKey (column)}
data-counts-copy={String (count (row, column))}
@@ -271,11 +274,13 @@ const CountsSummaryTable = (
data-counts-aggregate={(row.kind !== 'data' || column.kind !== 'data')
? '' : undefined}
className="border px-2 py-1 dark:border-slate-700">
<PrefetchLink to={drill (row, column)} aria-label={
{[...row.values, ...column.values].includes (countsMaskedValue)
? <span>{count (row, column)}</span>
: <PrefetchLink to={drill (row, column)} aria-label={
totalLabels.get (JSON.stringify ([row.values, column.values])) == null ? undefined
: `${message (totalLabels.get (
JSON.stringify ([row.values, column.values]))!).text}: ${count (row, column)}`}>
{count (row, column)}</PrefetchLink>
{count (row, column)}</PrefetchLink>}
</td>)}
</tr>)}</tbody>
</CountsSelectableTable>
@@ -288,7 +293,8 @@ const CountsSummaryResults = (props: Parameters<typeof CountsSummaryTable>[0]) =
const { message } = useUITranslation ('counts')
const headers = props.data.columnHeaders ?? []
const columns = props.data.cols ?? []
if (columns.length === 0) return <CountsSummaryTable {...props}/>
if (columns.length === 0 || props.data.columnValuesEditable === false)
return <CountsSummaryTable {...props}/>
const groups = [...new Set (headers.filter (header => header.values.length > 0)
.map (header => JSON.stringify (header.values.slice (0, 1))))]
const ids = groups.map ((_, index) => `group${index}`)
+106
ファイルの表示
@@ -0,0 +1,106 @@
import { fireEvent, screen } from '@testing-library/react'
import { describe, expect, it, vi } from 'vitest'
import CountsNestedTable from '@/components/counts/CountsNestedTable'
import CountsSelectableTable from '@/components/counts/CountsSelectableTable'
import CountsTableLayout from '@/components/counts/CountsTableLayout'
import { useCountsTableLayout } from '@/lib/countsTableLayout'
import { renderWithProviders } from '@/test/render'
import type { CountField } from '@/lib/counts'
const renderTable = () => renderWithProviders (<CountsTableLayout>
<CountsSelectableTable family="details" columns={['resource_id', 'version_no']}
identity="stable" name={{ text: 'Grid', language: 'en' }}>
<tbody>
<tr><td data-counts-copy="first cell"><span>first</span> <span>cell</span></td><td>second</td></tr>
<tr><td>third</td><td><input aria-label="editor" defaultValue="editable"/></td></tr>
</tbody>
</CountsSelectableTable>
</CountsTableLayout>, { route: '/ja/counts/posts' })
describe ('Counts cell keyboard contract', () => {
it ('moves Home/End on rows and Ctrl+Home/End on columns with a stable Shift anchor', () => {
renderTable ()
const table = screen.getByRole ('grid', { name: 'Grid' })
table.focus ()
fireEvent.keyDown (table, { key: 'End' })
expect (screen.getByText ('second')).toHaveAttribute ('data-counts-selected')
fireEvent.keyDown (table, { key: 'End', ctrlKey: true, shiftKey: true })
expect (screen.getByText ('second')).toHaveAttribute ('data-counts-selected')
expect (screen.getByRole ('textbox').closest ('td')).toHaveAttribute ('data-counts-selected')
expect (screen.getByText ('third')).not.toHaveAttribute ('data-counts-selected')
fireEvent.keyDown (table, { key: 'Home' })
expect (screen.getByText ('third')).toHaveAttribute ('data-counts-selected')
fireEvent.keyDown (table, { key: 'Home', ctrlKey: true })
expect (screen.getByText ('first').closest ('td')).toHaveAttribute ('data-counts-selected')
})
it.each (['Enter', 'Escape']) ('leaves text mode with %s and restores the cell focus', key => {
renderTable ()
const table = screen.getByRole ('grid', { name: 'Grid' })
const cell = screen.getByText ('first').closest ('td')!
table.focus ()
fireEvent.keyDown (table, { key: 'Home' })
fireEvent.keyDown (table, { key: 'Enter' })
expect (cell).toHaveAttribute ('data-counts-text')
expect (window.getSelection ()?.toString ()).toBe ('first cell')
const setData = vi.fn ()
fireEvent.copy (table, { clipboardData: { setData } })
expect (setData).not.toHaveBeenCalled ()
fireEvent.keyDown (table, { key })
expect (cell).not.toHaveAttribute ('data-counts-text')
expect (cell).toHaveAttribute ('data-counts-selected')
expect (window.getSelection ()?.toString ()).toBe ('')
expect (table).toHaveFocus ()
fireEvent.copy (table, { clipboardData: { setData } })
expect (setData).toHaveBeenCalledWith ('text/plain', 'first cell')
expect (setData).toHaveBeenCalledWith ('text/html', expect.stringContaining ('first cell'))
})
it ('does not consume input keyboard events and uses the sum of column widths', () => {
renderTable ()
const input = screen.getByRole ('textbox')
input.focus ()
expect (fireEvent.keyDown (input, { key: 'Enter' })).toBe (true)
expect (input).toHaveFocus ()
expect (screen.getByRole ('grid', { name: 'Grid' })).toHaveStyle ({ width: '176px' })
})
})
const WidthControls = () => {
const layout = useCountsTableLayout ()
return <>
<output>{layout.width ('details', 'title')}</output>
<button onClick={() => layout.fit ('details', 'title', 1000)}>fit</button>
<button onClick={() => layout.resize ('details', 'title', 180)}>resize</button>
</>
}
it ('caps automatic header sizing and preserves manual widths on subsequent fitting', () => {
renderWithProviders (<CountsTableLayout><WidthControls/></CountsTableLayout>)
fireEvent.click (screen.getByRole ('button', { name: 'fit' }))
expect (screen.getByRole ('status')).toHaveTextContent ('640')
fireEvent.click (screen.getByRole ('button', { name: 'resize' }))
fireEvent.click (screen.getByRole ('button', { name: 'fit' }))
expect (screen.getByRole ('status')).toHaveTextContent ('180')
})
it ('wraps nested JSON without truncating, reordering or changing its data', () => {
const fields: CountField[] = [
{ key: 'id', label: 'counts:columns.resourceId' },
{ key: 'name', label: 'counts:columns.name' }]
const rows = [{ id: 1, name: `https://example.invalid/${'a'.repeat (300)}` },
{ id: 2, name: 'line one\nline two "quoted"' }]
const params = new URLSearchParams ({ children: 'tags.id tags.name', nested_json: '1' })
const { container } = renderWithProviders (<CountsNestedTable rows={rows} fields={fields}
path="tags" resource="posts" title={{ text: 'Tags', language: 'en' }}
params={params} update={vi.fn ()}/>, { route: '/ja/counts/posts' })
const pre = container.querySelector ('pre')!
expect (pre.textContent).toBe (JSON.stringify (rows))
expect (pre).toHaveClass ('whitespace-pre-wrap', '[overflow-wrap:anywhere]', 'font-sans')
expect (pre).not.toHaveClass ('truncate', 'whitespace-pre', 'select-none')
})
+10
ファイルの表示
@@ -70,6 +70,16 @@ describe ('SidebarComponent', () => {
vi.spyOn (HTMLElement.prototype, 'hasPointerCapture').mockReturnValue (true)
})
it ('limits sidebar scrolling to desktop to avoid clipping mobile suggestions', () => {
renderSidebar ()
// jsdom has no layout; guard the responsive classes responsible for clipping.
const sidebar = screen.getByText ('Sidebar body')
expect (sidebar).not.toHaveClass ('overflow-y-auto')
expect (sidebar).toHaveClass ('md:overflow-y-auto')
})
it ('uses maxWidth without applying the container main-width reserve twice', async () => {
localStorage.setItem ('sidebar:test-sidebar:left:width', '700')
const onWidthChange = vi.fn ()
+1 -1
ファイルの表示
@@ -339,7 +339,7 @@ const SidebarComponent: FC<Props> = ({
</style>
</Helmet>
<div className="h-full overflow-y-auto p-4 sidebar">
<div className="h-full md:overflow-y-auto p-4 sidebar">
{children}
</div>
+3
ファイルの表示
@@ -141,9 +141,11 @@ export const render = async (
helmetContext = { }
requestContext.status = status
html = await renderDocument (page ())
// Client-only queries must neither run here nor enter the bootstrap below.
const pending = client.getQueryCache ().getAll ().filter (query => {
const options = query.options as QueryObserverOptions
return query.state.status === 'pending'
&& query.meta?.clientOnly !== true
&& options.queryFn != null && options.queryFn !== skipToken
&& options.enabled !== false
&& (typeof options.enabled !== 'function' || options.enabled (query))
@@ -173,6 +175,7 @@ export const render = async (
.map (entry => entry.toString ()).join ('')
const dehydrated = dehydrate (client, { shouldDehydrateQuery: query =>
query.state.status !== 'pending'
&& query.meta?.clientOnly !== true
&& !(query.queryKey[0] === 'help' && query.queryKey[1] === 'edition') })
const state = { ...dehydrated, queries: dehydrated.queries.map (query => ({
...query, state: { ...query.state, fetchFailureReason: null,
+1 -1
ファイルの表示
@@ -14,7 +14,7 @@
.ainu-ui-word
{
display: inline-block;
display: inline;
white-space: nowrap !important;
word-break: keep-all !important;
overflow-wrap: normal !important;
+36
ファイルの表示
@@ -162,4 +162,40 @@ describe ('api helpers', () => {
await apiPatch ('/posts/1')
expect (mocks.client.get).toHaveBeenCalledTimes (2)
})
it ('aborts one write while a shared CSRF lookup remains pending for another write', async () => {
let complete!: (value: { data: { token: string } }) => void
mocks.client.get.mockReturnValueOnce (new Promise (resolve => { complete = resolve }))
const { apiPost } = await import ('@/lib/api')
const controller = new AbortController ()
const first = apiPost ('/counts/export/jobs', {}, { signal: controller.signal })
const second = apiPost ('/posts', {})
const rejected = expect (first).rejects.toMatchObject ({ name: 'AbortError' })
controller.abort ()
await rejected
complete ({ data: { token: 'shared' } })
await second
expect (mocks.client.get).toHaveBeenCalledWith ('/users/csrf',
expect.objectContaining ({ timeout: 15_000 }))
expect (mocks.client.post).toHaveBeenCalledTimes (1)
expect (mocks.client.post.mock.calls[0][0]).toBe ('/posts')
})
it ('releases native download frames after failure or explicit cancellation', async () => {
const { startAPIDownload } = await import ('@/lib/api')
const failure = vi.fn ()
const release = startAPIDownload ('/counts/export', 'file', failure,
new URLSearchParams ({ job: 'job', attempt: 'attempt' }))
const frame = document.querySelector ('iframe')!
expect (new URL (frame.src).searchParams.get ('attempt')).toBe ('attempt')
frame.dispatchEvent (new Event ('error'))
frame.dispatchEvent (new Event ('error'))
expect (failure).toHaveBeenCalledTimes (1)
expect (frame).not.toBeInTheDocument ()
release ()
const cancel = startAPIDownload ('/counts/export', 'file', failure)
cancel ()
expect (document.querySelector ('iframe')).toBeNull ()
expect (failure).toHaveBeenCalledTimes (1)
})
})
+46 -11
ファイルの表示
@@ -63,7 +63,22 @@ const withSession = (opt?: Opt): AxiosRequestConfig => {
const withCSRF = async (opt?: Opt): Promise<AxiosRequestConfig> => {
const token = await csrfToken ()
const signal = opt?.signal
if (signal?.aborted) throw new DOMException ('Aborted', 'AbortError')
const pending = csrfToken ()
// A shared CSRF lookup must not make an individual aborted write wait forever.
const token = signal == null ? await pending : await new Promise<string> ((resolve, reject) => {
const abort = () => reject (new DOMException ('Aborted', 'AbortError'))
signal.addEventListener ('abort', abort, { once: true })
if (signal.aborted) abort ()
void pending.then (value => {
signal.removeEventListener ('abort', abort)
resolve (value)
}, error => {
signal.removeEventListener ('abort', abort)
reject (error)
})
})
const config = withSession (opt)
return { ...config, headers: { ...config.headers, 'X-CSRF-Token': token } }
}
@@ -73,7 +88,8 @@ const csrfToken = async (): Promise<string> => {
const state = csrfState ()
if (state.token != null)
return state.token
state.pending ??= client.get<{ token: string }> ('/users/csrf', withSession ())
state.pending ??= client.get<{ token: string }> (
'/users/csrf', { ...withSession (), timeout: 15_000 })
.then (response => {
if (csrfState () !== state)
return csrfToken ()
@@ -219,20 +235,39 @@ export const startAPIDownload = (
path: string,
token: string,
onError?: () => void,
): void => {
query?: URLSearchParams,
): (() => void) => {
const frame = document.createElement ('iframe')
frame.hidden = true
let timer: number | null = null
let closed = false
const release = () => {
if (closed) return
closed = true
if (timer != null) window.clearTimeout (timer)
frame.remove ()
}
const failure = () => {
if (closed) return
release ()
onError?.()
}
frame.addEventListener ('load', () => {
const document = frame.contentDocument
if (document != null && document.URL !== 'about:blank')
try
{
onError?.()
frame.remove ()
const content = frame.contentDocument
// Attachment navigations normally have no document. An error document is failure.
if (content != null && content.URL !== 'about:blank') failure ()
}
catch
{
// Cross-origin navigation cannot be inspected; the authenticated ACK decides.
}
})
frame.addEventListener ('error', () => { onError?.(); frame.remove () })
frame.src = `${API_BASE_URL}${path}/files/${encodeURIComponent (token)}`
frame.addEventListener ('error', failure)
const suffix = query == null ? '' : `?${query.toString ()}`
frame.src = `${API_BASE_URL}${path}/files/${encodeURIComponent (token)}${suffix}`
document.body.append (frame)
// Attachments do not reliably fire load; keep this isolated from page navigation.
window.setTimeout (() => frame.remove (), 10 * 60_000)
timer = window.setTimeout (release, 10 * 60_000)
return release
}
+191 -28
ファイルの表示
@@ -1,41 +1,204 @@
import { describe, expect, it } from 'vitest'
import { columnsForCounts, countColumns, countsCSV, normaliseCountsParams } from '@/lib/counts'
import { columnsForCounts, countColumns, countsCSV, countsDefinitionParams, countsResourcePath,
countsRequestParams, normaliseCountsParams, selectedCountChildren } from '@/lib/counts'
import { countsErrorMessage, countsConditionValue, countsNestedIsScalar, countsNestedScalarText }
from '@/lib/countsDisplay'
import { countsClipboard } from '@/lib/countsClipboard'
import { createUII18n, translateUIMessage } from '@/lib/i18n'
import { encodeCountState, readCountState, setCountState } from '@/lib/queryState'
import { isCountsPath } from '@/lib/routeLocales'
import { nameColumns, postDefinition } from '@/test/fixtures/counts'
import type { CountsCondition } from '@/lib/countsDefinitions'
describe ('counts exports', () => {
it ('resolves every output and CSV header key in both locales', () => {
for (const locale of ['ja', 'ain-Kana'])
{
const instance = createUII18n (locale)
for (const key of Object.values (countColumns))
{
const translated = translateUIMessage (key, 'counts', locale, { }, instance)
expect (translated.text).not.toBe (key)
expect (translated.text).not.toContain ('counts:')
}
const fallback = translateUIMessage ('run', 'counts', locale, { }, instance)
expect (fallback).toEqual ({ text: '集計実行', language: 'ja' })
}
describe ('Counts URL contract', () => {
it ('distinguishes HTTP, network and internal failures without server details', () => {
const http = { isAxiosError: true, response: { status: 503, data: '<private SQL>' } }
expect (countsErrorMessage (http, 'ja').text).toContain ('HTTP 503')
expect (countsErrorMessage (http, 'ja').text).not.toContain ('private SQL')
expect (countsErrorMessage ({ isAxiosError: true }, 'ja').text)
.toBe ('APIサーバーとの通信に失敗しました。')
expect (countsErrorMessage (new Error ('private stack'), 'ja').text)
.toBe ('内部処理で問題が発生しました。')
})
it ('explains a totals timeout using the server time limit', () => {
const failure = { isAxiosError: true, response: { status: 503,
data: { error: 'counts_query_timeout', stage: 'totals', timeout_seconds: 30 } } }
const text = countsErrorMessage (failure, 'ja').text
expect (text).toContain ('集計処理が制限時間を超えました。')
expect (text).toContain ('小計・総計の算出が30秒以内に完了しませんでした。')
expect (text).toContain ('集計期間や行・列の項目を減らして')
expect (countsErrorMessage (failure, 'ain-Kana').language).toBe ('ja')
})
it ('uses hyphenated page paths for internal resource identifiers', () => {
expect (countsResourcePath ('nico_tags')).toBe ('/counts/nico-tags')
expect (countsResourcePath ('tag_descriptions')).toBe ('/counts/tag-descriptions')
expect (countsResourcePath ('tag_groups')).toBe ('/counts/tag-groups')
expect (countsResourcePath ('posts')).toBe ('/counts/posts')
})
it ('defaults to latest summary, limit 100, and no axes', () => {
const params = normaliseCountsParams ('posts', new URLSearchParams (), postDefinition)
expect (params.get ('definition')).toBe ('latest')
expect (params.get ('mode')).toBe ('summary')
expect (params.get ('limit')).toBe ('100')
expect (params.get ('rows')).toBe ('')
expect (params.get ('cols')).toBe ('')
expect (params.has ('axis')).toBe (false)
expect (normaliseCountsParams ('posts', params, postDefinition).toString ())
.toBe (params.toString ())
})
it ('makes the monthly default explicit for the existing API', () => {
expect (normaliseCountsParams ('posts', new URLSearchParams ()).get ('axis'))
.toBe ('month')
it.each (['latest', 'history']) ('shares detail defaults for %s', definition => {
const params = normaliseCountsParams ('posts', new URLSearchParams ({
definition, mode: 'details' }), postDefinition)
expect (params.get ('sort')).toBe ('resource_id:asc version_no:asc')
expect ([...selectedCountChildren (params)]).toEqual (['tags.name_ja'])
expect (countsNestedIsScalar (nameColumns, 'tags', params)).toBe (true)
})
it ('escapes quotes, newlines and spreadsheet formulas in UTF-8 CSV', () => {
expect (countsCSV ([['日本語', 'a"b', 'a\nb', '=1+1']]))
.toBe ('\uFEFF"日本語","a""b","a\nb","\'=1+1"')
it.each (['20', '50', '100']) ('preserves an explicit limit of %s and empty selections', limit => {
const params = normaliseCountsParams ('posts', new URLSearchParams ({
mode: 'details', limit, sort: '', columns: '', children: '', rows: '', cols: '' }),
postDefinition)
for (const key of ['sort', 'columns', 'children', 'rows', 'cols'])
expect (params.get (key)).toBe ('')
expect (params.get ('limit')).toBe (limit)
})
it ('keeps resource-specific columns out of the other resource', () => {
expect (columnsForCounts ('posts')).toContain ('added_tags')
expect (columnsForCounts ('posts')).not.toContain ('aliases')
expect (columnsForCounts ('tags')).toContain ('aliases')
expect (columnsForCounts ('tags')).not.toContain ('url')
expect (columnsForCounts ('posts')).not.toContain ('event_type')
expect (columnsForCounts ('tags')).not.toContain ('event_type')
it ('normalises saved display keys without broadening script-specific predicates', () => {
const params = new URLSearchParams ({ mode: 'details', columns: 'tags',
children: 'tags.name_ain_kana tags.aliases_ja' })
setCountState (params, 'conditions', encodeCountState ('conditions', [
{ field: 'name_ain_kana', operator: 'eq', value: 'recorded' }]))
const result = normaliseCountsParams ('posts', params, postDefinition)
expect (result.get ('children')).toBe ('tags.name_ain tags.aliases_ja')
expect (result.get ('conditions.0.field')).toBe ('name_ain_kana')
expect (countsNestedIsScalar (nameColumns, 'tags', result)).toBe (false)
})
it ('unifies legacy drill conditions while keeping typed values, grain and execution time', () => {
const params = new URLSearchParams ({ execution_at: '2024-03-01T00:00:00Z',
timezone: 'Asia/Tokyo' })
const drills = [{ field: 'created_at', grain: 'month', value: '2024-02' },
{ field: 'user', value: null }, { field: 'resource_id', value: 7 },
{ field: 'flag', value: false }]
setCountState (params, 'drill', encodeCountState ('drill', drills))
const result = normaliseCountsParams ('posts', params)
expect (readCountState (result, 'conditions', []))
.toEqual (drills.map (entry => ({ ...entry, operator: 'axis_eq' })))
expect ([...result.keys ()].some (key => key === 'drill' || key.startsWith ('drill.')))
.toBe (false)
expect (result.get ('execution_at')).toBe (params.get ('execution_at'))
expect (result.get ('timezone')).toBe ('Asia/Tokyo')
setCountState (result, 'conditions', encodeCountState ('conditions', []))
expect (readCountState (result, 'conditions', [])).toEqual ([])
})
it ('round-trips free text and empty sets without JSON or delimiter splitting', () => {
const conditions: CountsCondition[] = [
{ field: 'title', operator: 'eq', value: 'a & b = c "quoted"' },
{ field: 'category', operator: 'in', input: 'list', values: [] },
{ field: 'version_no', operator: 'in', values: [1, 2] }]
const params = new URLSearchParams ()
setCountState (params, 'conditions', encodeCountState ('conditions', conditions))
expect (params.get ('conditions')).toBe ('')
expect (params.get ('conditions.0.value')).toBe (conditions[0].value)
expect (readCountState (params, 'conditions', [])).toEqual (conditions)
setCountState (params, 'conditions', encodeCountState ('conditions', [conditions[0]]))
expect ([...params.keys ()].some (key => key.startsWith ('condition_values'))).toBe (false)
})
it ('resets definition-dependent state but preserves timezone and row-number preference', () => {
const previous = new URLSearchParams ({ tags: 'alpha', rows: 'user', page: '4',
at: '2024-01-01', execution_at: '2024-02-01', timezone: 'Asia/Tokyo', row_numbers: '0' })
expect (Object.fromEntries (countsDefinitionParams ('history', previous))).toEqual ({
definition: 'history', mode: 'details', timezone: 'Asia/Tokyo', row_numbers: '0' })
})
it ('excludes presentation settings from API identity but retains extraction conditions', () => {
const params = new URLSearchParams ({ definition: 'history', columns: 'title',
children: 'tags.name_ja', order: 'title', nested_json: '1', row_numbers: '1',
resource_id: '7', limit: '50', sort: 'version_no:desc' })
expect (Object.fromEntries (countsRequestParams (params))).toEqual ({
definition: 'history', resource_id: '7', limit: '50', sort: 'version_no:desc' })
})
})
describe ('Counts logical display and exports', () => {
it ('resolves output labels in both supported locales', () => {
for (const locale of ['ja', 'ain-Kana'])
for (const key of Object.values (countColumns))
{
const message = translateUIMessage (key, 'counts', locale, {}, createUII18n (locale))
expect (message.text).not.toBe (key)
expect (message.text).not.toContain ('counts:')
}
expect (columnsForCounts ('posts')).toContain ('event_type')
expect (columnsForCounts ('tags')).not.toContain ('aliases')
})
it ('distinguishes unresolved, inaccessible and registered condition names', () => {
const axis = { field: 'tags' }
expect (countsConditionValue ('internal:7', axis, 'ja', 'UTC')).toBe ('…')
expect (countsConditionValue ('internal:7', axis, 'ja', 'UTC', { tags: {} })).toBe ('—')
expect (countsConditionValue ('internal:7', axis, 'ja', 'UTC', {
tags: { 'internal:7': 'recorded tag' } })).toBe ('recorded tag')
expect (countsConditionValue (7, { field: 'user' }, 'ja', 'UTC', {
user: { '7': 'actor' } })).toBe ('actor')
expect (countsConditionValue ('title', { field: 'changed_field' }, 'ja', 'UTC'))
.toBe (translateUIMessage ('columns.postTitle', 'counts', 'ja').text)
})
it ('formats calendar grain values without changing their timezone or identifier', () => {
expect (countsConditionValue ('2026', { field: 'created_at', grain: 'year' },
'ja', 'Asia/Tokyo')).toContain ('令和8年')
expect (countsConditionValue ('2026-10', { field: 'created_at', grain: 'month' },
'ja', 'America/New_York')).toContain ('令和8年10月')
})
it ('keeps all inner rows when choosing one display column', () => {
const params = new URLSearchParams ({ children: 'tags.name_ja' })
const rows = [{ name_ja: 'alpha' }, { name_ja: 'beta' }]
expect (countsNestedScalarText (rows, nameColumns, 'tags', params, 'ja')).toBe ('alpha beta')
expect (rows).toEqual ([{ name_ja: 'alpha' }, { name_ja: 'beta' }])
})
it ('escapes CSV and clipboard formulas, quotes, newlines and HTML', () => {
expect (countsCSV ([['日本語', 'a"b', 'a\nb', '=1+1']]))
.toBe ('\uFEFF"日本語","a""b","a\nb","\'=1+1"')
const copy = countsClipboard ([['<tag>', 'first\nsecond', '=1+1']])
expect (copy.text).toBe ('<tag>\t"first\nsecond"\t\'=1+1')
expect (copy.html).toContain ('&lt;tag&gt;')
expect (copy.html).toContain ('first<br>second')
expect (copy.html).toContain ('&#39;=1+1')
})
it.each (['/counts', '/counts/posts?q=1', '/ja/counts/tags', '/ain-Kana/counts/materials']) (
'classifies %s as excluded from search and speculative loading', path => {
expect (isCountsPath (path.split ('?')[0])).toBe (true)
})
})
describe ('Counts safe failure messages', () => {
it.each ([
[400, { error: 'too_many_axes', max_axes: 8 }, '最大8個'],
[422, { error: 'too_many_columns', max_columns: 73 }, '最大73列'],
[503, { error: 'counts_query_timeout' }, '時間制限'],
[503, { error: 'export_worker_unavailable' }, 'ワーカー'],
[400, {}, '入力条件'], [401, {}, '認証状態'], [403, {}, '権限']]) (
'distinguishes HTTP %s with a known safe response', (status, data, expected) => {
const failure = { isAxiosError: true, response: { status, data } }
expect (countsErrorMessage (failure, 'ja').text).toContain (expected)
})
it ('never renders arbitrary exception messages or private identifiers', () => {
const failure = { isAxiosError: true, response: { status: 500,
data: { error: 'SELECT private_title FROM secret_table', message: 'private-user-17' } } }
const text = countsErrorMessage (failure, 'ja').text
expect (text).not.toMatch (/SELECT|private|secret_table/)
expect (countsErrorMessage ({ error: 'worker_stalled' }, 'ja').text).toContain ('停止')
})
})
+25 -1
ファイルの表示
@@ -2,14 +2,20 @@ import { countsFieldLabel } from '@/lib/countsDefinitions'
import { apiDownload, apiGet } from '@/lib/api'
import { currentViewingLocale, localeLanguage } from '@/lib/locales'
import { readCountState, encodeCountState } from '@/lib/queryState'
import { readCountState, encodeCountState, setCountState } from '@/lib/queryState'
import type { CountsAxis, CountsDefinition } from '@/lib/countsDefinitions'
import type { LocaleCode } from '@/lib/locales'
import type { TagNameSnapshot } from '@/types'
export const countsMaskedValue = '__counts_private__'
export type CountsResource = 'posts' | 'tags' | 'nico_tags'
| 'tag_descriptions' | 'tag_groups' | 'materials'
export const countsResourcePath = (resource: CountsResource): string =>
`/counts/${resource.replaceAll ('_', '-')}`
export type CountsName = TagNameSnapshot & {
languageCode: string; languageName?: string | null; scriptName?: string | null }
export type CountsTag = {
@@ -77,6 +83,8 @@ export type CountsResult = {
executionAt?: string
columnValues?: (string | number | null)[][]
cells?: { values: (string | number | null)[]; count: number }[]
columnValuesEditable?: boolean
conditionLabels?: Record<string, Record<string, string | null>>
axisLabels?: Record<string, Record<string, string | null>>
axisNames?: Record<string, Record<string, Record<string, unknown>>>
rows?: { field: string; grain?: string }[]
@@ -368,6 +376,22 @@ export const normaliseCountsParams = (
definition?: CountsDefinition,
): URLSearchParams => {
const params = new URLSearchParams (source)
const drills = readCountState<Record<string, unknown>[]> (params, 'drill', [])
if (params.get ('tag'))
{
drills.push ({ field: 'tags', value: `${params.get ('kind') ?? 'internal'}:${
params.get ('tag')}` })
params.delete ('tag')
params.delete ('kind')
}
if (drills.length > 0)
{
const conditions = readCountState<Record<string, unknown>[]> (params, 'conditions', [])
setCountState (params, 'conditions', encodeCountState ('conditions', [
...conditions, ...drills.map (entry => ({ ...entry, operator: 'axis_eq' }))]))
for (const key of [...params.keys ()])
if (key === 'drill' || key.startsWith ('drill.')) params.delete (key)
}
if (!params.has ('limit')) params.set ('limit', '100')
const nameScope = params.get ('name_scope')
if (definition?.conditionAliases != null && (nameScope == null || nameScope === 'locale'))
+1 -1
ファイルの表示
@@ -8,7 +8,7 @@ import type { CountField, CountsResource } from '@/lib/counts'
export type CountsAxis = { field: string; grain?: string }
export type CountsCondition = {
field: string; operator: string; value?: string; end?: string; blank?: boolean
field: string; operator: string; grain?: string; value?: string | number | boolean | null; end?: string; blank?: boolean
input?: 'single' | 'multiple' | 'list'; values?: (string | number | boolean)[] }
export type CountsFieldType = 'string' | 'integer' | 'datetime' | 'enum'
| 'json' | 'boolean' | 'interval'
+79 -2
ファイルの表示
@@ -1,4 +1,5 @@
import { fieldsForCounts, orderedCountFields, selectedCountChildren,
import { isApiError } from '@/lib/api'
import { countsMaskedValue, fieldsForCounts, orderedCountFields, selectedCountChildren,
sortedCountChildren } from '@/lib/counts'
import { countsEventTypes, countsFieldLabel } from '@/lib/countsDefinitions'
import { languageUIMessage, translateUIMessage } from '@/lib/i18n'
@@ -12,6 +13,45 @@ import type { CountsAxis, CountsDefinition } from '@/lib/countsDefinitions'
import type { LocaleCode } from '@/lib/locales'
// Render only known codes and numerical limits, never arbitrary server error text.
export const countsErrorMessage = (failure: unknown, locale: LocaleCode) => {
const response = isApiError (failure) ? failure.response : null
const body = (response?.data ?? failure) as {
error?: unknown; stage?: unknown; timeout_seconds?: unknown; timeoutSeconds?: unknown
max_axes?: unknown; maxAxes?: unknown
max_columns?: unknown; maxColumns?: unknown } | null
const code = body != null && typeof body === 'object' ? body.error : null
const message = (key: string, values?: Record<string, string | number>) =>
translateUIMessage (key, 'counts', locale, values)
if (code === 'too_many_axes' || code === 'too_many_columns')
{
const maximum = code === 'too_many_axes' ? body?.maxAxes ?? body?.max_axes
: body?.maxColumns ?? body?.max_columns
if (typeof maximum === 'number')
return message (code === 'too_many_axes' ? 'axisLimit' : 'columnLimit.message', {
max: maximum })
}
if (code === 'private_condition') return message ('privateCondition')
if (code === 'counts_query_timeout' && body?.stage === 'totals')
{
const seconds = body.timeoutSeconds ?? body.timeout_seconds
if (typeof seconds === 'number') return message ('errors.totalsTimeout', { seconds })
}
if (code === 'counts_query_timeout' || code === 'generation_timeout')
return message ('errors.queryTimeout')
if (code === 'export_worker_unavailable' || code === 'worker_unavailable')
return message ('errors.workerUnavailable')
if (['worker_stalled', 'interrupted', 'generation_failed', 'export_too_large',
'export_storage_full'].includes (String (code))) return message ('errors.stopped')
if (response?.status === 401) return message ('errors.authentication')
if (response?.status === 403) return message ('errors.permission')
if (response?.status === 400 || response?.status === 422) return message ('errors.input')
if (response != null) return message ('errors.http', { status: response.status })
if (isApiError (failure)) return message ('errors.network')
return message ('errors.internal')
}
export const countsFieldValue = (row: CountRecord, field: string): unknown =>
row[field] ?? row[field.replace (/_([a-z])/g, (_, letter: string) => letter.toUpperCase ())]
@@ -126,7 +166,7 @@ export const countsLogicalName = (
if (value == null) return { text: '' }
if (field.startsWith ('aliases_') && Array.isArray (value))
return { text: value.map (entry => typeof entry === 'string' ? entry
: String (entry?.name ?? '')).join ('\n') }
: String (entry?.name ?? '')).join (' ') }
if (field === 'sections')
return { text: (countsSectionRanges (value, locale) ?? []).join (
translateUIMessage ('sections.separator', 'counts', locale).text) }
@@ -445,6 +485,40 @@ const countsDateValue = (
}
export const countsConditionValue = (
value: string | number | boolean | null | undefined,
axis: CountsAxis, locale: LocaleCode, timezone: string,
labels?: Record<string, Record<string, string | null>>,
): string => {
if (value === countsMaskedValue) return '********'
if (axis.field === 'user' && value == null)
return translateUIMessage ('bot', 'counts', locale).text
if (value == null) return 'NULL'
if (axis.field === 'tags' || axis.field === 'user')
{
if (labels == null) return '…'
const names = labels[axis.field]
if (names == null || !Object.prototype.hasOwnProperty.call (names, String (value)))
return '—'
return names[String (value)]
|| translateUIMessage ('notRegistered', 'counts', locale).text
}
if (axis.field === 'changed_field')
return translateUIMessage (countsFieldLabel (String (value)), 'counts', locale).text
if (typeof value === 'boolean')
return translateUIMessage (`truthValues.${value}`, 'counts', locale).text
if (axis.grain != null)
return countsDateValue (String (value), axis.grain, locale, timezone) ?? String (value)
if (axis.field === 'event_type')
return translateUIMessage (`events.${value}`, 'counts', locale).text
if (axis.field === 'category' || axis.field === 'tag_category')
return translateUIMessage (`common:categories.${value}`, 'counts', locale).text
if (['language_code', 'source_language_code'].includes (axis.field))
return languageUIMessage (String (value), locale).text
return String (value)
}
export const countsSummaryValue = (
value: string | number | null,
axis: CountsAxis,
@@ -455,12 +529,15 @@ export const countsSummaryValue = (
): string => {
const message = (key: string, values?: Record<string, string | number>) =>
translateUIMessage (key, 'counts', locale, values).text
if (value === countsMaskedValue) return '********'
if (value == null) return axis.field === 'user' ? message ('bot') : ''
if (['language_code', 'source_language_code'].includes (axis.field))
return languageUIMessage (String (value), locale).text
const logical = data.axisLabels?.[`a${index}`]?.[String (value)]
if (axis.field === 'user') return message ('actorWithId', {
name: logical || message ('notRegistered'), id: value })
if (axis.field === 'parent_tag_ids')
return String (value) === '' ? '' : logical || message ('notRegistered')
if (logical != null) return logical
if (axis.field === 'tags')
return message ('notRegistered')
+2 -2
ファイルの表示
@@ -1,4 +1,4 @@
import { isCountsState, selectedCountFields } from '@/lib/counts'
import { countsResourcePath, isCountsState, selectedCountFields } from '@/lib/counts'
import { translateUIMessage } from '@/lib/i18n'
import type { CountField, CountsResource, CountsResult } from '@/lib/counts'
@@ -90,5 +90,5 @@ export const countsSummaryDrill = (
next.set ('before',
(before != null && Date.parse (before) < Date.parse (last)) ? before : last)
}
return `/counts/${resource}?${next}`
return `${countsResourcePath (resource)}?${next}`
}
+2 -2
ファイルの表示
@@ -28,7 +28,7 @@ describe ('locale URL generation', () => {
'/tos', '/gekanator', '/more']) ('localises and redirects %s', pathname => {
const location = { pathname, search: '?page=2&tag=a%23b', hash: '#section' }
expect (localiseTo (pathname, 'ain-Kana')).toBe (`/ain-Kana${ pathname }`)
expect (legacyLocalePath (location)).toBe (`/ja${ pathname }${ location.search }#section`)
expect (legacyLocalePath (location, () => true)).toBe (`/ja${ pathname }${ location.search }#section`)
expect (unlocalisedPath (`/ain-Kana${ pathname }`)).toBe (pathname)
})
@@ -50,6 +50,6 @@ describe ('locale URL generation', () => {
it.each (['/en/posts', '/ain/posts', '/unknown', '/api/tags', '/assets/image.png']) (
'does not redirect invalid or non-page paths: %s', pathname => {
expect (legacyLocalePath ({ pathname, search: '', hash: '' })).toBeNull ()
expect (legacyLocalePath ({ pathname, search: '', hash: '' }, () => false)).toBeNull ()
})
})
+17 -1
ファイルの表示
@@ -1,6 +1,7 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { fetchPostChanges, fetchPosts, toggleViewedFlg, updatePost } from '@/lib/posts'
import { fetchPostChanges, fetchPosts, switchPostFilterLocale, toggleViewedFlg, updatePost }
from '@/lib/posts'
import type { FetchPostsParams } from '@/types'
@@ -34,6 +35,21 @@ describe ('posts API functions', () => {
vi.clearAllMocks ()
})
it ('switches current tag names without adding identity state to the URL', async () => {
api.apiGet.mockResolvedValueOnce ({ tags: 'シロシ not:nico:shared' })
const path = await switchPostFilterLocale ({
location: { pathname: '/ja/counts/posts',
search: '?tags=source+not%3Anico%3Ashared&match=any', hash: '#table' },
params: {}, locale: 'ja', targetLocale: 'ain-Kana' })
expect (api.apiGet).toHaveBeenCalledWith ('/posts/tag_filters', {
locale: 'ja', params: { tags: 'source not:nico:shared', target_locale: 'ain-Kana' } })
const url = new URL (path, 'https://example.test')
expect (url.pathname).toBe ('/ain-Kana/counts/posts')
expect (Object.fromEntries (url.searchParams)).toEqual ({
tags: 'シロシ not:nico:shared', match: 'any' })
expect (url.hash).toBe ('#table')
})
it ('maps post search parameters to backend snake_case names', async () => {
api.apiGet.mockResolvedValueOnce ({ posts: [], count: 0 })
+4 -18
ファイルの表示
@@ -1,5 +1,4 @@
import { apiDelete, apiGet, apiPost, apiPut } from '@/lib/api'
import { queryRecordParams, setQueryRecords } from '@/lib/queryState'
import { switchLocalePath } from '@/lib/locales'
import type { LocaleCode } from '@/lib/locales'
@@ -8,7 +7,7 @@ import type { FetchPostsParams, Post, PostVersion } from '@/types'
export const fetchPosts = async (
{ url, title, tags, tagReferences, match, createdFrom, createdTo, updatedFrom, updatedTo,
{ url, title, tags, match, createdFrom, createdTo, updatedFrom, updatedTo,
originalCreatedFrom, originalCreatedTo, page, limit, order }: FetchPostsParams,
locale: LocaleCode = 'ja',
): Promise<{
@@ -18,7 +17,6 @@ export const fetchPosts = async (
...(url && { url }),
...(title && { title }),
...(tags && { tags }),
...(tagReferences && queryRecordParams (new URLSearchParams (tagReferences), 'tag_refs')),
...(match && { match }),
...(createdFrom && { created_from: createdFrom }),
...(createdTo && { created_to: createdTo }),
@@ -39,21 +37,9 @@ export const switchPostFilterLocale = async (
if (tags == null || tags.trim () === '')
return switchLocalePath (location, targetLocale)
const { references } = await apiGet<{ references: {
name: string
tagId?: number
externalTagId?: number
negative?: boolean
missing?: boolean }[] }> ('/posts/tag_filters', {
locale, params: { tags, target_locale: targetLocale,
...queryRecordParams (query, 'tag_refs') } })
query.set ('tags', references.map (reference => reference.name).join (' '))
setQueryRecords (query, 'tag_refs', references.map (reference => ({
name: reference.name,
negative: reference.negative,
...(reference.tagId != null && { tag_id: reference.tagId }),
...(reference.externalTagId != null && { external_tag_id: reference.externalTagId }),
...(reference.missing && { missing: true }) })))
const result = await apiGet<{ tags: string }> ('/posts/tag_filters', {
locale, params: { tags, target_locale: targetLocale } })
query.set ('tags', result.tags)
return switchLocalePath ({ ...location, search: `?${query}` }, targetLocale)
}

変更されたファイルが多すぎるため、一部のファイルは表示されません さらに表示