コミットを比較

...
14 コミット
作成者 SHA1 メッセージ 日付
みてるぞ f91f9d0eb5 #434 2026-10-26 12:44:22 +09:00
みてるぞ 3e8f209542 #434 2026-10-26 12:19:43 +09:00
みてるぞ 8642225b94 #434 2026-10-25 23:08:48 +09:00
みてるぞ 96a5bb8601 #434 2026-10-25 22:16:20 +09:00
みてるぞ e71663bd22 #434 2026-10-25 16:47:05 +09:00
みてるぞ 52a8d62be5 #434 2026-10-25 16:25:04 +09:00
みてるぞ 3e05757820 #434 2026-10-25 16:00:37 +09:00
みてるぞ d411b3b40f #434 2026-10-25 14:53:35 +09:00
みてるぞ 304090f717 #434 2026-10-25 14:14:55 +09:00
みてるぞ 084a090cab アイヌ語追加 (#428) (#433)
Reviewed-on: #433
Co-authored-by: miteruzo <miteruzo@naver.com>
2026-10-20 20:44:14 +09:00
みてるぞ 75d7dac999 Wiki 依存の排除 (#431) (#432)
Reviewed-on: #432
Co-authored-by: miteruzo <miteruzo@naver.com>
2026-10-16 00:26:53 +09:00
みてるぞ e44887cf15 ヘルプを Wiki から分離 (#424) (#430)
画面上でのテストは完了.
移行にす早さが求められるので,準備ができたらマージする.

Reviewed-on: #430
Co-authored-by: miteruzo <miteruzo@naver.com>
2026-10-13 12:25:11 +09:00
みてるぞ 534a66f90d タグ・グループを Wiki から分離 (#425) (#429)
Reviewed-on: #429
Co-authored-by: miteruzo <miteruzo@naver.com>
2026-10-11 03:39:47 +09:00
みてるぞ 592932b5a5 ニコニコ同期の環境変数修正 (#426) (#427)
Reviewed-on: #427
2026-10-07 04:26:04 +09:00
412個のファイルの変更、27536行の追加、5694行の削除
+17 -4
ファイルの表示
@@ -573,10 +573,10 @@ and layout reuse, follow `frontend/AGENTS.md`.
- When new user-facing wording appears necessary, ask the user for the exact
wording and placement before implementing it.
- Do not invent replacement copy when removing unrequested wording.
- Do not create, modify, or run tests unless the user explicitly asks for
test work. When the user asks for tests, keep working within the permitted
test-file scope and rerun them until they pass or the remaining failure is
clearly blocked.
- 明示的な個別指示がない限り、テストの追加・修正・削除・実行を一切行わない。
fixture、mock、snapshot、テスト補助コードも対象とする。
既存のテスト差分を勝手に修正・差戻ししない。
この規約はディレクトリ固有の自動的なテスト実行推奨より優先する。
- Test-only work includes adding, updating, deleting, reorganising, or fixing
SyntaxError in tests. During test-only work, do not modify production code.
- During test-only work, do not change production constants, behaviour, API
@@ -631,6 +631,19 @@ and layout reuse, follow `frontend/AGENTS.md`.
## Frontend rules
- 原則として、ユーザ向け UI 文言を TSX / TypeScript に直接記述しない。
新しい文言は既存の i18n システムで管理し、適切な既存キーを先に確認する。
不必要な重複定義を避け、新規機能固有の文言は適切な namespace にまとめる。
- 対応 locale の辞書を整合させる。未翻訳項目には既存の日本語 fallback と
`_untranslated` 規約を使い、language metadata を維持する。
翻訳を捏造せず、未翻訳文字列を正式な翻訳として登録しない。
- UI 文言の追加・変更時はビルド、静的検査、実画面の確認等で、辞書と
SSR / hydration の整合及び翻訳キーの非露出を検証する。
i18n の検証を理由にテスト関連作業を開始しない。
- API の内部識別子、パラメータ、ユーザ入力データ、外部 URL は機械的に
翻訳しない。ボタン、項目名、CSV ヘッダ、エラー、空状態、ツールチップ、
アクセシビリティ用ラベルも UI 文言として扱う。
- Use `frontend/src/lib/api.ts` for API calls so headers and camelCase conversion stay consistent.
- Add or reuse TanStack Query keys through `frontend/src/lib/queryKeys.ts`;
avoid ad hoc query key arrays.
+48 -2
ファイルの表示
@@ -1,22 +1,52 @@
class ApplicationController < ActionController::API
include ActionController::Cookies
include ActionController::RequestForgeryProtection
class UnsupportedLocale < StandardError
;
end
rescue_from ActiveRecord::RecordInvalid, with: :render_record_invalid
rescue_from ActiveRecord::RecordNotUnique, with: :render_record_not_unique
rescue_from(UnsupportedLocale) { head :not_found }
before_action :reject_banned_ip_address!
before_action :authenticate_user
before_action :reject_banned_user!
before_action :validate_explicit_locale!
protect_from_forgery with: :exception
self.forgery_protection_origin_check = true
rescue_from ActionController::InvalidAuthenticityToken, with: :reject_invalid_csrf
after_action :prevent_authenticated_caching
def current_user = @current_user
private
def reject_invalid_csrf
response.headers['X-CSRF-Invalid'] = '1'
response.headers['Cache-Control'] = 'private, no-store'
head :forbidden
end
def authenticate_user
code = request.headers['X-Transfer-Code'] || request.headers['HTTP_X_TRANSFER_CODE']
code = session[:inheritance_code]
return if code.blank?
@current_user = User.find_by(inheritance_code: code)
end
def prevent_authenticated_caching
response.headers['Cache-Control'] = 'private, no-store' if current_user
end
def establish_user_session user
reset_session
session[:inheritance_code] = user.inheritance_code
@current_user = user
response.headers['Cache-Control'] = 'private, no-store'
end
def bool? key, default: false
return default if params[key].nil?
@@ -93,5 +123,21 @@ class ApplicationController < ActionController::API
value
end
def resolve_locale! = Locale.find_by(code: params[:locale]) || Locale.nipponese
def resolve_locale!
return @resolved_locale if defined?(@resolved_locale)
code = params.fetch(:locale, 'ja')
unless code.is_a?(String) && Locale.codes.value?(code)
raise UnsupportedLocale
end
locale = Locale.find_by(code:)
raise UnsupportedLocale unless locale && locale.deprecated_at.nil?
@resolved_locale = locale
end
def validate_explicit_locale!
resolve_locale! if params.key?(:locale)
end
end
+21
ファイルの表示
@@ -0,0 +1,21 @@
module CountsListing
extend ActiveSupport::Concern
def index
locale = resolve_locale!
resource = is_a?(CountsPostsController) ? 'posts' : 'tags'
query = CountsQuery.new(version_scope, resource, request.query_parameters.except('locale')
.merge('locale' => locale.code))
result = { count: query.count, page: query.page, limit: query.limit,
at: request.query_parameters['at'].presence || Time.current.utc.iso8601(6) }
if query.mode == 'summary'
result.merge!(query.summary)
else
result[:versions] = CountsVersionRepr.many(
query.details, resource:, language_code: locale.language_code)
end
render json: result
rescue ArgumentError, TypeError
head :bad_request
end
end
+45
ファイルの表示
@@ -0,0 +1,45 @@
class CountsExportsController < ApplicationController
wrap_parameters false
def prepare
locale = resolve_locale!
payload = request.request_parameters.merge(request.query_parameters)
exporter = CountsExporter.new(payload, language_code: locale.language_code)
session[:counts_export_owner] ||= SecureRandom.hex(32)
token = CountsExportDownload.prepare(exporter, session[:counts_export_owner])
response.headers['Cache-Control'] = 'private, no-store'
render json: { token: }
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
head :bad_request
end
def download
file = CountsExportDownload.find(params[:token], session[:counts_export_owner])
return head :not_found unless file
response.headers['Cache-Control'] = 'private, no-store'
send_file file.fetch('path'),
filename: "counts-#{ file.fetch('resource') }.#{ file.fetch('format') }",
type: file['format'] == 'json' ? 'application/json' : 'text/csv',
disposition: 'attachment'
end
def index
locale = resolve_locale!
payload = request.request_parameters.except('authenticity_token')
.merge(request.query_parameters)
exporter = CountsExporter.new(payload,
language_code: locale.language_code)
response.headers['Content-Type'] = exporter.format == 'json' ?
'application/json; charset=utf-8' : 'text/csv; charset=utf-8'
response.headers['Content-Disposition'] =
"attachment; filename=\"counts-#{ exporter.resource }.#{ exporter.format }\""
response.headers['Cache-Control'] = 'private, no-store'
response.headers['X-Accel-Buffering'] = 'no'
# Rack::ETag otherwise materialises the response to calculate a digest.
response.headers['Last-Modified'] = Time.current.httpdate
self.response_body = exporter.body
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
head :bad_request
end
end
+3
ファイルの表示
@@ -0,0 +1,3 @@
class CountsPostsController < PostVersionsController
include CountsListing
end
+10
ファイルの表示
@@ -0,0 +1,10 @@
class CountsStatesController < ApplicationController
def index
locale = resolve_locale!
result = CountsStateQuery.new(request.query_parameters,
language_code: locale.language_code).call
render json: result
rescue ArgumentError, TypeError
head :bad_request
end
end
+21
ファイルの表示
@@ -0,0 +1,21 @@
class CountsTablesController < ApplicationController
def definition
resolve_locale!
definition = CountsResourceDefinition.new(params[:resource])
render json: { fields: definition.metadata,
axes: { latest: definition.axes(false), history: definition.axes(true) },
grains: CountsAxes::GRAINS }
rescue ArgumentError
head :bad_request
end
def index
resolve_locale!
render json: CountsTableQuery.new(request.query_parameters).call
rescue CountsTableQuery::TooManyColumns
render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS },
status: :unprocessable_entity
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
head :bad_request
end
end
+3
ファイルの表示
@@ -0,0 +1,3 @@
class CountsTagsController < TagVersionsController
include CountsListing
end
+10
ファイルの表示
@@ -0,0 +1,10 @@
class LocalesController < ApplicationController
def index
locales = Locale.where(deprecated_at: nil).order(:created_at).map do |locale|
{ code: locale.code_before_type_cast, name: locale.name }
end
response.headers['Cache-Control'] = 'no-store'
render json: { locales: }
end
end
+6 -2
ファイルの表示
@@ -70,7 +70,10 @@ class MaterialsController < ApplicationController
.find_by(id: params[:id])
return head :not_found unless material
wiki_page_body = material.tag&.tag_name('ja')&.wiki_page&.current_revision&.body
wiki_page_body =
TagDescription
.find_by(tag_id: material.tag_id, language_code: locale.language_code)
&.body
render json: MaterialRepr.base(material, host: request.base_url,
language_code: locale.language_code).merge(wiki_page_body:)
@@ -252,7 +255,8 @@ class MaterialsController < ApplicationController
end
def resolve_material_tag! locale, tag_name_raw
Tag.find_or_create_by_tag_name!(locale, tag_name_raw, category: :material)
Tag.find_or_create_by_tag_name!(locale, tag_name_raw, category: :material,
created_by_user: current_user)
end
def material_index_needs_tag_name? filters
+23 -9
ファイルの表示
@@ -24,9 +24,9 @@ class NicoTagsController < ApplicationController
q =
ExternalTag
.joins("LEFT JOIN (#{ post_tag_max_sql }) post_tag_max " \
.joins("LEFT JOIN (#{ post_tag_max_sql }) post_tag_max " +
'ON post_tag_max.external_tag_id = external_tags.id')
.includes(linked_tags: { tag_names: :wiki_page })
.includes(linked_tags: [:tag_names, :tag_descriptions])
if name
q = q.where(('external_tags.name LIKE ? ' +
"OR CONCAT(external_tags.platform, ':', external_tags.name) LIKE ?"),
@@ -37,7 +37,7 @@ class NicoTagsController < ApplicationController
linked_tag_ids =
Tag
.joins(:tag_names)
.where(tag_names: { language_code: 'ja', primary_flg: true })
.where(tag_names: { language_code: locale.language_code, primary_flg: true })
.where('tag_names.name LIKE ?', "%#{ linked_tag }%")
.pluck(:id)
linked_nico_tag_ids = NicoTagRelation.where(tag_id: linked_tag_ids).pluck(:nico_tag_id)
@@ -45,7 +45,7 @@ class NicoTagsController < ApplicationController
end
if link_status.in?(['linked', 'unlinked'])
exists_sql =
'EXISTS (SELECT 1 FROM nico_tag_relations ' \
'EXISTS (SELECT 1 FROM nico_tag_relations ' +
'WHERE nico_tag_relations.nico_tag_id = external_tags.id)'
q = link_status == 'linked' ? q.where(exists_sql) : q.where("NOT #{ exists_sql }")
end
@@ -98,7 +98,8 @@ class NicoTagsController < ApplicationController
ApplicationRecord.transaction do
linked_tags = Tag.normalise_tags!(locale, linked_tag_names,
with_tagme: false,
with_no_deerjikist: false)
with_no_deerjikist: false,
created_by_user: current_user)
TagVersioning.record_tag_snapshots!(
linked_tags,
@@ -140,10 +141,23 @@ class NicoTagsController < ApplicationController
end
def render_nico_tag_form_record_invalid record
if record.is_a?(TagName) || record.is_a?(Tag)
render_validation_error fields: { tags: record.errors.full_messages.map { |message|
"タグ名 “#{ record.name }”: #{ message }"
} }
names =
case record
when TagName
[record]
when Tag
record.tag_names.to_a
else
[]
end
messages = names.flat_map do |tag_name|
tag_name.errors.full_messages.map do |message|
"タグ名 “#{ tag_name.name }”: #{ message }"
end
end
if messages.any?
render_validation_error fields: { tags: messages }
else
render_validation_error record
end
+46 -27
ファイルの表示
@@ -1,5 +1,6 @@
class PostVersionsController < ApplicationController
def index
locale = resolve_locale!
post_id = params[:post].presence
tag_id = params[:tag].presence&.to_i
external_tag_id = params[:external_tag].presence&.to_i
@@ -13,21 +14,7 @@ class PostVersionsController < ApplicationController
offset = (page - 1) * limit
q = PostVersion.joins(<<~SQL.squish)
LEFT JOIN
post_versions prev
ON
prev.post_id = post_versions.post_id
AND prev.version_no = post_versions.version_no - 1
SQL
.select('post_versions.*',
'prev.title AS prev_title',
'prev.url AS prev_url',
'prev.thumbnail_base AS prev_thumbnail_base',
'prev.tags_json AS prev_tags_json',
'prev.video_ms AS prev_video_ms',
'prev.original_created_from AS prev_original_created_from',
'prev.original_created_before AS prev_original_created_before')
q = version_scope
q = q.where('post_versions.post_id = ?', post_id) if post_id
if external_tag_id || (tag_id && !(Tag.exists?(id: tag_id)))
q = q.where('JSON_CONTAINS(post_versions.tags_json,' +
@@ -45,12 +32,24 @@ class PostVersionsController < ApplicationController
.limit(limit)
.offset(offset)
render json: { versions: serialise_versions(versions), count: }
render json: { versions: serialise_versions(versions, locale.language_code), count: }
end
private
def serialise_versions rows
def version_scope
PostVersion.with_previous_version
.select('post_versions.*',
'prev.title AS prev_title',
'prev.url AS prev_url',
'prev.thumbnail_base AS prev_thumbnail_base',
'prev.tags_json AS prev_tags_json',
'prev.video_ms AS prev_video_ms',
'prev.original_created_from AS prev_original_created_from',
'prev.original_created_before AS prev_original_created_before')
end
def serialise_versions rows, language_code
rows = rows.to_a
user_ids = rows.map(&:created_by_user_id).compact.uniq
users_by_id = User.where(id: user_ids).pluck(:id, :name).to_h
@@ -58,12 +57,16 @@ class PostVersionsController < ApplicationController
[normalise_json(row.tags_json),
normalise_json(row.attributes['prev_tags_json']) || []]
}
tag_ids = snapshots.flatten.filter_map { _1['tag_id'] }.uniq
tag_names = TagName.where(tag_id: tag_ids, language_code:, primary_flg: true)
.pluck(:tag_id, :name).to_h
external_tag_names = external_tag_names_for(snapshots)
rows.map do |row|
cur_tags = snapshot_tag_literals(normalise_json(row.tags_json), external_tag_names)
cur_tags = snapshot_tag_literals(
normalise_json(row.tags_json), tag_names, external_tag_names)
prev_tags = snapshot_tag_literals(
normalise_json(row.attributes['prev_tags_json']) || [], external_tag_names)
normalise_json(row.attributes['prev_tags_json']) || [], tag_names, external_tag_names)
{ post_id: row.post_id,
version_no: row.version_no,
@@ -75,6 +78,7 @@ class PostVersionsController < ApplicationController
prev: row.attributes['prev_thumbnail_base'] },
video_ms: { current: row.video_ms, prev: row.attributes['prev_video_ms'] },
tags: build_version_tags(cur_tags, prev_tags),
parent_post_ids: row.parent_post_ids,
original_created_from: {
current: row.original_created_from&.iso8601,
prev: row.attributes['prev_original_created_from']&.iso8601 },
@@ -106,17 +110,27 @@ class PostVersionsController < ApplicationController
names
end
def snapshot_tag_literals snapshots, external_tag_names
def snapshot_tag_literals snapshots, tag_names, external_tag_names
snapshots
.filter_map { |snapshot|
if snapshot.key?('tag_id')
sections = snapshot.fetch('sections', []).map { |section|
[section.fetch('begin_ms'), section['end_ms']]
}.sort_by { |begin_ms, end_ms| [begin_ms, end_ms || Float::INFINITY] }
name = tag_names[snapshot['tag_id']] || snapshot.fetch('name')
[tag_category_order(snapshot['category']),
Post.tag_snapshot_literal(snapshot)]
{ identity: [:tag, snapshot['tag_id'], sections],
name: Post.tag_snapshot_literal(snapshot.merge('name' => name)),
restore_name: tag_names[snapshot['tag_id']] &&
Post.tag_snapshot_literal(snapshot.merge('name' => name)),
external: false }]
elsif external_tag_names[snapshot['external_tag_id']]
[6, external_tag_names[snapshot['external_tag_id']]]
[6, { identity: [:external_tag, snapshot['external_tag_id']],
name: external_tag_names[snapshot['external_tag_id']],
restore_name: nil, external: true }]
end
}
.sort_by { |order, name| [order, name.downcase] }
.sort_by { |order, tag| [order, tag[:name].downcase] }
.map(&:second)
end
@@ -125,17 +139,22 @@ class PostVersionsController < ApplicationController
end
def build_version_tags cur_tags, prev_tags
(cur_tags | prev_tags).map do |name|
current = cur_tags.index_by { _1[:identity] }
previous = prev_tags.index_by { _1[:identity] }
(current.keys | previous.keys).map do |identity|
type =
if cur_tags.include?(name) && prev_tags.include?(name)
if current.key?(identity) && previous.key?(identity)
'context'
elsif cur_tags.include?(name)
elsif current.key?(identity)
'added'
else
'removed'
end
{ name:, type: }
entry = current[identity] || previous[identity]
{ name: entry[:name], restore_name: entry[:restore_name],
external: entry[:external], type: }
end
end
end
+71 -26
ファイルの表示
@@ -53,7 +53,7 @@ class PostsController < ApplicationController
.reselect('posts.*', Arel.sql("#{ updated_at_all_sql } AS updated_at_all"))
.preload(:external_tags, :uploaded_user, :parents, :children,
post_tags: [:sections, { tag: [:deerjikists, :materials,
{ tag_names: :wiki_page }] }])
:tag_names, :tag_descriptions] }])
.with_attached_thumbnail
q = q.where('posts.url LIKE ?', "%#{ url }%") if url
@@ -111,7 +111,7 @@ class PostsController < ApplicationController
filtered_posts(locale)
.preload(:uploaded_user, :parents, :children,
post_tags: [:sections, { tag: [:deerjikists, :materials,
{ tag_names: :wiki_page }] }])
:tag_names, :tag_descriptions] }])
.with_attached_thumbnail
.order('RAND()')
.first
@@ -198,7 +198,7 @@ class PostsController < ApplicationController
Post
.includes(:uploaded_user, :parents, :children,
post_tags: [:sections, { tag: [:deerjikists, :materials,
{ tag_names: :wiki_page }] }])
:tag_names, :tag_descriptions] }])
.with_attached_thumbnail
.find_by(id: params[:id])
return head :not_found unless post
@@ -400,8 +400,60 @@ class PostsController < ApplicationController
render_post_form_record_invalid e.record
end
def tag_filters
source = resolve_locale!
code = params[:target_locale]
return head :not_found unless code.is_a?(String) && Locale.codes.value?(code)
target = Locale.find_by(code:, deprecated_at: nil)
return head :not_found unless target
literals = params[:tags].to_s.split
references = tag_filter_references(literals)
names = literals.each_with_index.map do |literal, index|
reference = references&.[](index)
(reference && reference['negative'] == false) ? literal : literal.sub(/\Anot:/i, '')
end
internal = references ? {} : TagName.where(
language_code: source.language_code, name: names).pluck(:name, :tag_id).to_h
canonical = references ? {} : TagName.where(
language_code: source.language_code, primary_flg: true,
tag_id: internal.values).pluck(:tag_id, :name).to_h
external_names = names.map { canonical[internal[_1]] || _1 }
external = references ? {} : ExternalTag.where(
"CONCAT(platform, ':', name) IN (?)", external_names).pluck(
Arel.sql("CONCAT(platform, ':', name)"), :id).to_h
ids = references ? references.filter_map { _1['tag_id'] } : internal.values
translated = TagName.where(language_code: target.language_code,
primary_flg: true, tag_id: ids).pluck(:tag_id, :name).to_h
filters = literals.each_with_index.map do |literal, index|
name = names[index]
reference = if references
references[index].except('name')
else
matches = {}
matches['tag_id'] = internal[name] if internal[name]
external_id = external[external_names[index]]
matches['external_tag_id'] = external_id if external_id
matches.empty? ? { 'missing' => true } : matches
end
name = translated[reference['tag_id']] || name
negative = reference.fetch('negative', literal.downcase.start_with?('not:'))
prefix = negative ? literal[/\Anot:/i] || 'not:' : ''
reference.merge('name' => "#{ prefix }#{ name }", 'negative' => negative)
end
render json: { references: filters }
end
private
def tag_filter_references literals
PostTagSearch.parse_references(params[:tag_refs], literals)
rescue ArgumentError, JSON::ParserError, TypeError
raise ActionController::BadRequest
end
def filtered_posts locale
tag_names = params[:tags].to_s.split
match_type = params[:match]
@@ -413,20 +465,19 @@ class PostsController < ApplicationController
end
def filter_posts_by_tags locale, tag_names, match_type
literals = tag_names.map do |raw_name|
{ name: TagName.canonicalise(locale, raw_name.sub(/\Anot:/i, '')).first,
negative: raw_name.downcase.start_with?('not:') }
end
references = tag_filter_references(tag_names)
literals = PostTagSearch.literals(locale, tag_names.join(' '), references:)
return Post.all if literals.empty?
if match_type == 'any'
literals.reduce(Post.none) do |posts, literal|
posts.or(tag_literal_relation(locale, literal[:name], negative: literal[:negative]))
posts.or(tag_literal_relation(locale, literal[:name], negative: literal[:negative],
reference: literal[:reference]))
end
else
literals.reduce(Post.all) do |posts, literal|
ids = tagged_post_ids_for(locale, literal[:name])
ids = tagged_post_ids_for(locale, literal[:name], reference: literal[:reference])
if literal[:negative]
posts.where.not(id: ids)
else
@@ -436,8 +487,8 @@ class PostsController < ApplicationController
end
end
def tag_literal_relation locale, name, negative:
ids = tagged_post_ids_for(locale, name)
def tag_literal_relation locale, name, negative:, reference: nil
ids = tagged_post_ids_for(locale, name, reference:)
if negative
Post.where.not(id: ids)
else
@@ -445,19 +496,11 @@ class PostsController < ApplicationController
end
end
def tagged_post_ids_for locale, name
posts_by_internal_tags =
Post
.joins(tags: :tag_names)
.where(tag_names: { language_code: locale.language_code, name: })
.select(:id)
posts_by_external_tags =
Post
.joins(:external_tags)
.where("CONCAT(external_tags.platform, ':', external_tags.name) = ?", name)
(posts_by_internal_tags + posts_by_external_tags).uniq(&:id)
def tagged_post_ids_for locale, name, reference: nil
internal, external = PostTagSearch.references(locale, { name:, reference: })
Post.where(id: PostTag.where(tag_id: Array(internal)).select(:post_id))
.or(Post.where(id: PostExternalTag.where(external_tag_id: Array(external))
.select(:post_id))).select(:id)
end
def sync_post_tags! post, desired_tags, sections
@@ -728,7 +771,8 @@ class PostsController < ApplicationController
with_tagme: false,
with_no_deerjikist: false,
deny_deprecated: true,
with_sections: true) => { tags:, sections: }
with_sections: true,
created_by_user: current_user) => { tags:, sections: }
tags = Tag.expand_parent_tags(tags).reject(&:deprecated?)
video_ms = normalise_video_ms(tags, video_ms_param:, duration_param:)
@@ -874,7 +918,8 @@ class PostsController < ApplicationController
Tag.normalise_tags!(locale, snapshot[:tag_names],
with_tagme: false,
deny_deprecated: true,
with_sections: true) => { tags:, sections: }
with_sections: true,
created_by_user: current_user) => { tags:, sections: }
TagVersioning.record_tag_snapshots!(tags, created_by_user: current_user,
language_code: locale.language_code)
+57
ファイルの表示
@@ -0,0 +1,57 @@
class SitemapEntriesController < ApplicationController
PAGE_SIZE = 1_000
RESOURCES = [
'posts', 'tags', 'deerjikists', 'materials',
'descriptions', 'tag_groups', 'theatres'].freeze
def index
locale = resolve_locale!
resources = RESOURCES.map { |resource| { resource:, count: entries(resource, locale).count } }
render json: { resources:, page_size: PAGE_SIZE }
end
def show
locale = resolve_locale!
resource = params[:resource]
return head :not_found unless RESOURCES.include?(resource)
page = Integer(params.fetch(:page, '1'), exception: false)
return head :bad_request unless page && page.positive?
scope = entries(resource, locale)
return head :not_found if page > [(scope.count.to_f / PAGE_SIZE).ceil, 1].max
ids = scope.order(:id).offset((page - 1) * PAGE_SIZE).limit(PAGE_SIZE).pluck(:id)
render json: { ids: }
end
private
def entries resource, locale
named_tags = TagName.where(language_code: locale.language_code, primary_flg: true)
.select(:tag_id)
case resource
when 'posts'
Post.all
when 'tags'
Tag.where(id: named_tags)
when 'deerjikists'
Tag.where(id: named_tags).where(id: Deerjikist.select(:tag_id))
when 'materials'
Material.all
when 'descriptions'
TagDescription.where(language_code: locale.language_code, tag_id: named_tags)
.where.not(body: [nil, ''])
.where(TagDescriptionVersion
.where('tag_description_versions.tag_id = tag_descriptions.tag_id')
.where('tag_description_versions.language_code = tag_descriptions.language_code')
.where('tag_description_versions.version_no = tag_descriptions.version_no')
.arel.exists)
when 'tag_groups'
TagGroup.joins(:versions)
.where('tag_group_versions.version_no = tag_groups.version_no')
when 'theatres'
Theatre.kept
end
end
end
+12 -7
ファイルの表示
@@ -1,14 +1,18 @@
class TagDescriptionsController < ApplicationController
def show
description = TagDescription.find_by(tag_id: params[:id], language_code: 'ja')
locale = resolve_locale!
description = TagDescription.find_by(tag_id: params[:id], language_code: locale.language_code)
return head :not_found unless description
render json: representation(description)
end
def show_by_title
tag_name = TagName.find_by(language_code: 'ja', name: params[:name].to_s.strip)
description = tag_name&.tag && TagDescription.find_by(tag: tag_name.tag, language_code: 'ja')
locale = resolve_locale!
tag_name = TagName.find_by(
language_code: locale.language_code, name: params[:name].to_s.strip)
description = tag_name&.tag && TagDescription.find_by(
tag: tag_name.tag, language_code: locale.language_code)
return head :not_found unless description
render json: representation(description)
@@ -21,18 +25,19 @@ class TagDescriptionsController < ApplicationController
body = params[:body].to_s
return render_unprocessable_entity('本文は必須です.', field: :body) if body.blank?
locale = resolve_locale!
tag = Tag.find_by(id: params[:id])
return head :not_found unless tag
description = TagDescription.find_by(tag:, language_code: 'ja')
description = TagDescription.find_by(tag:, language_code: locale.language_code)
if description
TagDescriptionOperation.update!(tag:, body:, updated_by_user: current_user)
TagDescriptionOperation.update!(locale, tag:, body:, updated_by_user: current_user)
description.reload
render json: representation(description)
else
TagDescriptionOperation.create!(tag:, body:, created_by_user: current_user)
description = TagDescription.find_by!(tag:, language_code: 'ja')
TagDescriptionOperation.create!(locale, tag:, body:, created_by_user: current_user)
description = TagDescription.find_by!(tag:, language_code: locale.language_code)
render json: representation(description), status: :created
end
rescue ActiveRecord::RecordInvalid => e
+43
ファイルの表示
@@ -0,0 +1,43 @@
class TagGroupsController < ApplicationController
def index
locale = resolve_locale!
groups = TagGroup.includes(:created_by_user, :updated_by_user).order(:id)
render json: TagGroupRepr.many(locale, groups.to_a)
end
def show
locale = resolve_locale!
group = TagGroup.includes(:created_by_user, :updated_by_user).find_by(id: params[:id])
return head :not_found unless group
render json: TagGroupRepr.one(locale, group)
end
def create
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
locale = resolve_locale!
group = TagGroupOperation.create!(locale, **write_attributes, actor: current_user)
render json: TagGroupRepr.one(locale, group), status: :created
end
def update
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
locale = resolve_locale!
group = TagGroup.find_by(id: params[:id])
return head :not_found unless group
TagGroupOperation.update!(locale, group:, **write_attributes, actor: current_user)
render json: TagGroupRepr.one(locale, group.reload)
end
private
def write_attributes
{ name: params[:name], description: params[:description],
members: params[:members], placements: params[:placements] }
end
end
+67 -21
ファイルの表示
@@ -10,17 +10,7 @@ class TagVersionsController < ApplicationController
offset = (page - 1) * limit
q = TagVersion.joins(<<~SQL.squish)
LEFT JOIN
tag_versions prev
ON
prev.tag_id = tag_versions.tag_id
AND prev.version_no = tag_versions.version_no - 1
SQL
.select('tag_versions.*', 'prev.name AS prev_name', 'prev.category AS prev_category',
'prev.deprecated_at AS prev_deprecated_at',
'prev.aliases AS prev_aliases', 'prev.names AS prev_names',
'prev.parent_tag_ids AS prev_parent_tag_ids')
q = version_scope
q = q.where('tag_versions.tag_id = ?', tag_id) if tag_id
count = q.except(:select, :order, :limit, :offset).count
@@ -34,9 +24,38 @@ class TagVersionsController < ApplicationController
private
def version_scope
TagVersion.joins(<<~SQL.squish)
LEFT JOIN
tag_versions prev
ON
prev.tag_id = tag_versions.tag_id
AND prev.version_no = tag_versions.version_no - 1
SQL
.select('tag_versions.*', 'prev.name AS prev_name', 'prev.category AS prev_category',
'prev.deprecated_at AS prev_deprecated_at',
'prev.aliases AS prev_aliases', 'prev.names AS prev_names',
'prev.parent_tag_ids AS prev_parent_tag_ids')
end
def serialise_versions rows, language_code
rows = rows.to_a
tag_ids = rows.map(&:tag_id).uniq
current_names_by_tag_id = TagName.where(tag_id: tag_ids, language_code:, primary_flg: true)
.pluck(:tag_id, :name, :auto_generated).to_h do |id, name, automatic|
[id, { name:, auto_generated: automatic }]
end
user_ids = rows.map(&:created_by_user_id).compact.uniq
users_by_id = User.where(id: user_ids).pluck(:id, :name).to_h
parent_ids = rows.flat_map do |row|
split_parent_tag_ids(row.parent_tag_ids) +
split_parent_tag_ids(row.attributes['prev_parent_tag_ids'])
end.uniq
parents_by_id = Tag.includes(
:tag_names, :materials, :deerjikists, :tag_descriptions,
parents: [:tag_names, :materials, :deerjikists, :tag_descriptions])
.where(id: parent_ids).index_by(&:id)
parent_json = parents_by_id.transform_values { TagRepr.base(_1, language_code:) }
rows.map do |row|
current_names = names_for_locale(row.names, language_code)
@@ -47,18 +66,10 @@ class TagVersionsController < ApplicationController
cur_parent_tag_ids = split_parent_tag_ids(row.parent_tag_ids)
prev_parent_tag_ids = split_parent_tag_ids(row.attributes['prev_parent_tag_ids'])
all_parent_tag_ids = (cur_parent_tag_ids | prev_parent_tag_ids)
tags_by_id =
Tag
.includes(:tag_names, :materials, tag_names: :wiki_page)
.where(id: all_parent_tag_ids)
.index_by(&:id)
parent_tags =
build_version_values(cur_parent_tag_ids, prev_parent_tag_ids, key: :tag_id)
.map do |h|
{ tag: TagRepr.base(tags_by_id[h[:tag_id]], language_code:),
{ tag: parent_json.fetch(h[:tag_id]),
type: h[:type] }
end
@@ -66,6 +77,8 @@ class TagVersionsController < ApplicationController
version_no: row.version_no,
event_type: row.event_type,
names: row.names,
name_changes: name_changes(row.names, row.attributes['prev_names']),
current_name_reference: current_names_by_tag_id[row.tag_id],
name: { current: current_names['name'], prev: previous_names['name'] },
category: { current: row.category, prev: row.attributes['prev_category'] },
deprecated_at: { current: row.deprecated_at&.iso8601,
@@ -97,11 +110,44 @@ class TagVersionsController < ApplicationController
def split_values(values) = values.to_s.split(/\s+/).reject(&:blank?)
def names_for_locale names, language_code
values = snapshot_names(names)[language_code]
values.is_a?(Hash) ? values : {}
end
def snapshot_names names
values = names.is_a?(String) ? JSON.parse(names) : names
values&.fetch(language_code, {}) || {}
values.is_a?(Hash) ? values : {}
rescue JSON::ParserError
{}
end
def name_changes current_names, previous_names
current = snapshot_names(current_names)
previous = snapshot_names(previous_names)
(current.keys | previous.keys).sort.filter_map do |language|
current_name = language_snapshot(current[language])
previous_name = language_snapshot(previous[language])
next if current_name == previous_name
{ language_code: language, current: current_name, prev: previous_name }
end
end
def language_snapshot entry
return nil unless entry.is_a?(Hash)
aliases = Array(entry['aliases']).filter_map do |value|
value = { 'name' => value } if value.is_a?(String)
next unless value.is_a?(Hash) && value['name'].is_a?(String)
{ name: value['name'], script_code: value['script_code'] }
end
aliases = aliases.uniq.sort_by { [_1[:name], _1[:script_code].to_s] }
{ name: entry['name'],
script_code: entry['script_code'],
auto_generated: entry['auto_generated'],
aliases: }
end
def split_parent_tag_ids(values) = split_values(values).map(&:to_i)
end
+37 -45
ファイルの表示
@@ -175,7 +175,8 @@ class TagsController < ApplicationController
tags_by_id =
Tag
.joins(:tag_names)
.includes(:tag_names, :materials, tag_names: :wiki_page)
.preload(:tag_names, :materials, :deerjikists, :tag_descriptions,
parents: [:tag_names, :materials, :deerjikists, :tag_descriptions])
.where(tag_names: { language_code: locale.language_code, primary_flg: true },
id: tag_ids)
.index_by(&:id)
@@ -238,7 +239,8 @@ class TagsController < ApplicationController
base =
Tag
.joins(:tag_names)
.includes(:tag_names, :materials, tag_names: :wiki_page)
.preload(:tag_names, :materials, :deerjikists, :tag_descriptions,
parents: [:tag_names, :materials, :deerjikists, :tag_descriptions])
.where(tag_names: { language_code: locale.language_code, primary_flg: true },
deprecated_at: nil)
base = base.where('tags.post_count > 0') if present_only
@@ -282,7 +284,8 @@ class TagsController < ApplicationController
tag =
Tag
.joins(:tag_names)
.includes(:tag_names, :materials, tag_names: :wiki_page)
.preload(:tag_names, :materials, :deerjikists, :tag_descriptions,
parents: [:tag_names, :materials, :deerjikists, :tag_descriptions])
.find_by(id: params[:id],
tag_names: { language_code: locale.language_code, primary_flg: true })
return render json: TagRepr.base(tag, language_code: locale.language_code) if tag
@@ -302,8 +305,9 @@ class TagsController < ApplicationController
tag =
Tag
.joins(:tag_names)
.includes(:tag_names, :materials, tag_names: :wiki_page)
.find_by(tag_names: { name:, language_code: locale.language_code, primary_flg: true })
.preload(:tag_names, :materials, :deerjikists, :tag_descriptions,
parents: [:tag_names, :materials, :deerjikists, :tag_descriptions])
.find_by(tag_names: { name:, language_code: locale.language_code })
return render json: TagRepr.base(tag, language_code: locale.language_code) if tag
platform, external_name = name.split(':', 2)
@@ -320,7 +324,8 @@ class TagsController < ApplicationController
tag =
Tag.joins(:tag_names)
.includes(:tag_names, tag_names: :wiki_page)
.preload(:tag_names, :materials, :deerjikists, :tag_descriptions,
parents: [:tag_names, :materials, :deerjikists, :tag_descriptions])
.find_by(id: params[:id],
tag_names: { language_code: locale.language_code, primary_flg: true })
return head :not_found unless tag
@@ -340,7 +345,8 @@ class TagsController < ApplicationController
tag =
Tag
.joins(:tag_names)
.includes(:tag_names, tag_names: :wiki_page)
.preload(:tag_names, :materials, :deerjikists, :tag_descriptions,
parents: [:tag_names, :materials, :deerjikists, :tag_descriptions])
.find_by(tag_names: { name:, language_code: locale.language_code, primary_flg: true })
return head :not_found unless tag
@@ -359,7 +365,8 @@ class TagsController < ApplicationController
tag =
Tag
.joins(:tag_names)
.includes(:tag_names, tag_names: :wiki_page)
.preload(:tag_names, :materials, :deerjikists, :tag_descriptions,
parents: [:tag_names, :materials, :deerjikists, :tag_descriptions])
.find_by(id: params[:id],
tag_names: { language_code: locale.language_code, primary_flg: true })
return head :not_found unless tag
@@ -368,6 +375,7 @@ class TagsController < ApplicationController
return if performed?
ApplicationRecord.transaction do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
tag.lock!
requested_keys = rows.map { |row| [row[:platform], row[:code]] }.uniq
@@ -424,7 +432,8 @@ class TagsController < ApplicationController
tag =
Tag
.joins(:tag_names)
.includes(:tag_names, :materials, tag_names: :wiki_page)
.preload(:tag_names, :materials, :deerjikists, :tag_descriptions,
parents: [:tag_names, :materials, :deerjikists, :tag_descriptions])
.find_by(tag_names: { name:, language_code: locale.language_code, primary_flg: true })
return head :not_found unless tag
@@ -770,31 +779,15 @@ class TagsController < ApplicationController
TagVersioning.ensure_snapshot!(old_owner_tag, created_by_user: current_user)
end
current_tag_name.wiki_page&.update!(tag_name_id: promoted_tag_name.id)
current_tag_name.association(:wiki_page).reset
current_tag_name.update!(canonical_id: promoted_tag_name.id,
primary_flg: false, auto_generated: false)
promoted_tag_name.update!(tag:, canonical_id: nil, primary_flg: true, auto_generated: false)
TagName.where(tag:,
language_code: current_tag_name.language_code,
primary_flg: false)
.where.not(id: promoted_tag_name.id)
.find_each do |alias_tag_name|
alias_tag_name.update!(canonical_id: promoted_tag_name.id, primary_flg: false)
end
tag.update!(tag_name_id: promoted_tag_name.id)
current_tag_name.association(:wiki_page).reset
current_tag_name.association(:tag).reset
current_tag_name.reload.update!(canonical_id: promoted_tag_name.id, primary_flg: false)
current_tag_name&.update!(primary_flg: false, auto_generated: false)
promoted_tag_name.update!(tag:, primary_flg: true, auto_generated: false)
tag.association(:tag_names).reset
return unless old_owner_tag && old_owner_tag != tag
record_tag_version!(old_owner_tag.reload, event_type: :update,
created_by_user: current_user,
language_code: current_tag_name.language_code)
language_code: promoted_tag_name.language_code)
end
def update_aliases! locale, tag, alias_names
@@ -802,20 +795,15 @@ class TagsController < ApplicationController
affected_tags = [tag]
primary_tag_name = TagName.find_by!(tag:,
language_code: locale.language_code,
primary_flg: true)
current_aliases = tag.tag_names.where(language_code: locale.language_code,
primary_flg: false).to_a
current_aliases.each do |alias_tag_name|
next if alias_names.include?(alias_tag_name.name)
affected_tags << alias_tag_name.tag
end
requested_aliases = TagName.where(
language_code: locale.language_code, name: alias_names)
.includes(:tag).index_by(&:name)
alias_names.each do |alias_name|
alias_tag_name = TagName.find_by(language_code: locale.language_code, name: alias_name)
alias_tag_name = requested_aliases[alias_name]
if alias_tag_name&.primary? && alias_tag_name.tag_id && alias_tag_name.tag_id != tag.id
alias_tag_name.errors.add(:name, 'その名前は既に使はれてゐます.')
raise ActiveRecord::RecordInvalid, alias_tag_name
@@ -823,7 +811,8 @@ class TagsController < ApplicationController
affected_tags << alias_tag_name&.tag
end
affected_tags.compact.uniq.each do |affected_tag|
affected_tags = affected_tags.compact.uniq(&:id)
affected_tags.each do |affected_tag|
TagVersioning.ensure_snapshot!(affected_tag, created_by_user: current_user,
language_code: locale.language_code)
end
@@ -831,19 +820,21 @@ class TagsController < ApplicationController
current_aliases.each do |alias_tag_name|
next if alias_names.include?(alias_tag_name.name)
alias_tag_name.update!(canonical_id: nil, tag_id: nil)
alias_tag_name.destroy!
end
alias_names.each do |alias_name|
alias_tag_name = TagName.find_or_initialize_by(
alias_tag_name = requested_aliases[alias_name] || TagName.new(
language_code: locale.language_code,
name: alias_name)
alias_tag_name.script_code = locale.script_code
alias_tag_name.update!(tag:, primary_flg: false, auto_generated: false,
canonical_id: primary_tag_name.id)
alias_tag_name.update!(tag:, primary_flg: false, auto_generated: false)
end
affected_tags.compact.uniq.each do |affected_tag|
affected_tags.each do |affected_tag|
# update_all records this Tag after its parent relationships are updated.
next if affected_tag.id == tag.id
record_tag_version!(affected_tag, event_type: :update,
created_by_user: current_user,
language_code: locale.language_code)
@@ -853,7 +844,8 @@ class TagsController < ApplicationController
def update_parent_tags! locale, tag, parent_names
parent_tags = Tag.normalise_tags!(locale, parent_names,
with_tagme: false,
with_no_deerjikist: false)
with_no_deerjikist: false,
created_by_user: current_user)
old_parent_tags = tag.parents.to_a
+33 -3
ファイルの表示
@@ -1,39 +1,69 @@
class UsersController < ApplicationController
def create
if current_user
return render json: { code: current_user.inheritance_code,
user: current_user.slice(:id, :name, :inheritance_code, :role) }
end
user = nil
User.transaction do
user = User.create!(inheritance_code: SecureRandom.uuid, role: :guest)
attach_ip_address!(user)
end
establish_user_session(user)
render json: { code: user.inheritance_code,
user: user.slice(:id, :name, :inheritance_code, :role) },
status: :created
end
def verify
user = User.find_by(inheritance_code: params[:code])
user = if bool?(:restore) && current_user
current_user
else
User.find_by(inheritance_code: params[:code])
end
return render json: { valid: false } unless user
return head :forbidden if user.banned?
attach_ip_address!(user)
establish_user_session(user)
render json: { valid: true, user: user.slice(:id, :name, :inheritance_code, :role) }
end
def show_session
return head :unauthorized unless current_user
render json: current_user.slice(:id, :name, :role)
end
def csrf
response.headers['Cache-Control'] = 'private, no-store'
render json: { token: form_authenticity_token }
end
def destroy_session
reset_session
response.headers['Cache-Control'] = 'private, no-store'
head :no_content
end
def renew
return head :unauthorized unless current_user
user = current_user
user.inheritance_code = SecureRandom.uuid
user.save!
establish_user_session(user)
render json: { code: user.inheritance_code }
end
def me
user = User.find_by(inheritance_code: params[:code])
return head :not_found unless user
user = current_user
return head :unauthorized unless user
attach_ip_address!(user)
render json: user.slice(:id, :name, :inheritance_code, :role)
end
+278 -146
ファイルの表示
@@ -1,107 +1,159 @@
class WikiPagesController < ApplicationController
rescue_from Wiki::Commit::Conflict, with: :render_wiki_conflict
# 互換性のための Facade API
rescue_from TagDescriptionOperation::Conflict, with: :render_wiki_conflict
def index
# 互換性のための化石リソース
# TODO: ヘルプおよびタグ・グループ追加後に,
# TODO: タグ説明とヘルプ、タグ・グループから取得する作りに変更する.
#
title = params[:title].to_s.strip
descriptions = TagDescription.where(language_code: 'ja')
.includes(tag: :tag_names)
pages = unmigrated_wiki_pages.includes(tag_name: :tag)
locale = resolve_locale!
title = params[:title].to_s.strip
descriptions =
TagDescription
.where(language_code: locale.language_code)
.includes(tag: :tag_names)
descriptions = descriptions.order(updated_at: :desc, id: :asc)
tag_groups = TagGroup.order(updated_at: :desc, id: :asc)
if title.present?
pattern = "%#{ ApplicationRecord.sanitize_sql_like(title) }%"
descriptions = descriptions.joins(tag: :tag_names)
.where(tag_names: { language_code: 'ja', primary_flg: true })
descriptions =
descriptions.joins(tag: :tag_names)
.where(tag_names: { language_code: locale.language_code, primary_flg: true })
.where('tag_names.name LIKE ?', pattern).limit(20)
pages = pages.where('tag_names.name LIKE ?', pattern).limit(20)
tag_groups = tag_groups.where(
'CONCAT(?, name) LIKE ? OR CONCAT(?, name) LIKE ?',
TagGroupRepr::PREFIX, pattern, 'tag_group:', pattern).limit(20)
end
json = descriptions.map { |description| WikiPageRepr.tag_description(description) }
json.concat(WikiPageRepr.base(pages))
json = descriptions.map { |description| WikiPageRepr.tag_description(locale, description) }
json.concat(tag_groups.map { WikiPageRepr.tag_group(_1) })
json.sort_by! do |row|
[-row['updated_at'].to_f, row.fetch('resource_type'), row.fetch('id')]
end
json = json.first(20) if title.present?
render json:
end
def autolink_candidates
locale = resolve_locale!
language = params.fetch(:language, locale.language_code)
language = 'ja' if language == 'und'
return head :not_found unless language.is_a?(String)
return head :not_found unless Language.exists?(code: language)
names = TagDescription.joins(tag: :tag_names)
.where(language_code: locale.language_code, tag_names: { language_code: language })
.pluck('tag_descriptions.tag_id', 'tag_descriptions.id', 'tag_names.name')
candidates = names.map do |tag_id, description_id, name|
WikiPageRepr.autolink_tag(locale, tag_id:, description_id:, name:, language:)
end
candidates.concat(TagGroup.order(:id).pluck(:id, :name).map do |id, name|
WikiPageRepr.autolink_group(locale, id:, name:, language:)
end)
render json: candidates
end
def show
tag_description = TagDescription.find_by(id: params[:id])
locale = resolve_locale!
reference = TagDescription.find_by(id: params[:id])
tag_description = reference && TagDescription.find_by(
tag_id: reference.tag_id, language_code: locale.language_code)
return render_wiki_page_by_tag_description tag_description if tag_description
return render_wiki_page_by_tag_description locale, tag_description if tag_description
return head :not_found if reference
page = WikiPage.joins(:tag_name)
.includes(tag_name: :tag)
.find_by(id: params[:id])
tag_group = TagGroup.find_by(id: params[:id])
return render_tag_group locale, tag_group if tag_group
render_wiki_page_or_404 page
head :not_found
end
def show_by_title
locale = resolve_locale!
title = params[:title].to_s.strip
tag_description = tag_description_by_title(title)
if (name = TagGroupRepr.name_from_title(title))
tag_group = TagGroup.find_by(name:)
return render_tag_group locale, tag_group if tag_group
return head :not_found
end
return render_wiki_page_by_tag_description tag_description if tag_description
tag_description = tag_description_by_title(locale, title)
page = WikiPage.joins(:tag_name)
.includes(tag_name: :tag)
.find_by(tag_name: { name: title })
return render_wiki_page_by_tag_description locale, tag_description if tag_description
render_wiki_page_or_404 page
head :not_found
end
def tag_reference
locale = resolve_locale!
description = TagDescription.find_by(id: params[:id])
return head :not_found unless description
tag = Tag.preload(:tag_names, :materials, :deerjikists, :tag_descriptions,
parents: [:tag_names, :materials, :deerjikists, :tag_descriptions])
.find(description.tag_id)
render json: TagRepr.base(tag, language_code: locale.language_code)
end
def exists
return head :no_content if TagDescription.exists?(id: params[:id])
locale = resolve_locale!
if TagDescription.exists?(id: params[:id], language_code: locale.language_code)
return head :no_content
end
return head :no_content if TagGroup.exists?(id: params[:id])
head WikiPage.exists?(id: params[:id]) ? :no_content : :not_found
head :not_found
end
def exists_by_title
locale = resolve_locale!
title = params[:title].to_s.strip
return head :no_content if tag_description_by_title(title)
if (name = TagGroupRepr.name_from_title(title))
exists = TagGroup.exists?(name:)
return head exists ? :no_content : :not_found
end
exists = WikiPage.joins(:tag_name).exists?(tag_names: { name: title })
head exists ? :no_content : :not_found
return head :no_content if tag_description_by_title(locale, title)
head :not_found
end
def diff
id = params[:id]
return render_bad_request('id は必須です.') if id.blank?
locale = resolve_locale!
from = params[:from].presence
to = params[:to].presence
description = TagDescription.find_by(id:)
description = TagDescription.find_by(id:, language_code: locale.language_code)
if description
versions = description_versions(description)
from_rev = from && versions.find(from)
to_rev = to ? versions.find(to) : versions.find_by!(version_no: description.version_no)
resource = WikiPageRepr.tag_description(description)
resource = WikiPageRepr.tag_description(locale, description)
else
page = WikiPage.joins(:tag_name).includes(tag_name: :tag).find(id)
from_rev = from && page.wiki_revisions.find(from)
to_rev = to ? page.wiki_revisions.find(to) : page.current_revision
if ((from_rev && !from_rev.content?) || !to_rev&.content?)
return render_unprocessable_entity('差分を表示できない版です.')
tag_group = TagGroup.find_by(id:)
if tag_group
from_rev = from && group_version(tag_group, from)
to_rev = to ? group_version(tag_group, to) : current_group_version(tag_group)
return head :not_found if (from && !from_rev) || !to_rev
from_body = from_rev && render_group_version(locale, from_rev)
to_body = render_group_version(locale, to_rev)
resource = { 'id' => tag_group.id,
'title' => "#{ TagGroupRepr::PREFIX }#{ tag_group.name }",
'deprecated_at' => nil }
else
return head :not_found
end
resource = WikiPageRepr.base(page)
end
diffs = Diff::LCS.sdiff(from_rev&.body&.lines || [], to_rev.body.lines)
diff_json = diffs.map { |change|
case change.action
when ?=
{ type: 'context', content: change.old_element }
when ?!
[{ type: 'removed', content: change.old_element },
{ type: 'added', content: change.new_element }]
when ?+
{ type: 'added', content: change.new_element }
when ?-
{ type: 'removed', content: change.old_element }
end
}.flatten.compact
from_body ||= from_rev&.body
to_body ||= to_rev.body
diff_json = wiki_diff(from_body, to_body)
render json: { wiki_page_id: resource.fetch('id'),
title: resource.fetch('title'),
@@ -109,6 +161,8 @@ class WikiPagesController < ApplicationController
older_revision_id: from_rev&.id,
newer_revision_id: to_rev.id,
diff: diff_json }
rescue TagGroupMarkdownRenderer::UnsupportedStructure => error
render_unprocessable_entity(error.message)
end
def create
@@ -122,30 +176,37 @@ class WikiPagesController < ApplicationController
return render_unprocessable_entity('タイトルは必須です.', field: :title) if title.blank?
return render_unprocessable_entity('本文は必須です.', field: :body) if body.blank?
page = if title.include?(':')
ApplicationRecord.transaction do
tag_name = TagName.find_or_create_by!(language_code: 'ja', name: title) do |name|
name.script_code = 'Jpan'
name.primary_flg = true
name.auto_generated = false
end
locale = resolve_locale!
Wiki::Commit.create_legacy_content!(
tag_name:, body:, created_by_user: current_user, message:)
end
else
tag_name = TagName.find_by(language_code: 'ja', name: title)
return render_unprocessable_entity field: :title unless tag_name&.tag
TagDescriptionOperation.create!(
tag: tag_name.tag, body:, created_by_user: current_user, message:)
if (name = TagGroupRepr.name_from_title(title))
parsed = TagGroupMarkdownParser.parse(locale, body:, group: nil)
group = TagGroupOperation.create!(
locale, name:, **parsed.except(:compatibility_body), actor: current_user,
require_compatibility: true)
version = current_group_version(group)
return render_group_representation(TagGroupRepr.one(locale, group), version,
status: :created)
end
render json: WikiPageRepr.base(page), status: :created
description =
if title.include?(':')
return render_unprocessable_entity field: :title
else
tag_name = TagName.find_by(language_code: locale.language_code, name: title)
return render_unprocessable_entity field: :title unless tag_name&.tag
TagDescriptionOperation.create!(
locale,
tag: tag_name.tag, body:, created_by_user: current_user, message:)
end
render json: WikiPageRepr.tag_description(locale, description), status: :created
rescue ActiveRecord::RecordInvalid => e
render_validation_error e.record
rescue ActiveRecord::RecordNotUnique
render_record_not_unique
rescue TagGroupMarkdownParser::InvalidDocument => error
render_unprocessable_entity(error.message)
end
def update
@@ -158,93 +219,178 @@ class WikiPagesController < ApplicationController
return render_unprocessable_entity('タイトルは必須です.', field: :title) if title.blank?
return render_unprocessable_entity('本文は必須です.', field: :body) if body.blank?
page = WikiPage.find(params[:id])
locale = resolve_locale!
group = TagGroup.find_by(id: params[:id])
if (name = TagGroupRepr.name_from_title(title))
return head :not_found unless group
parsed = TagGroupMarkdownParser.parse(locale, body:, group:)
TagGroupOperation.update!(
locale, group:, name:, **parsed.except(:compatibility_body), actor: current_user,
require_compatibility: true)
return head :ok
end
return render_unprocessable_entity field: :title if title.include?(':')
description = TagDescription.find_by(id: params[:id], language_code: locale.language_code)
unless description
return render_unprocessable_entity field: :title if group
return head :not_found
end
tag = description.tag
tag_name = tag.tag_name(locale.language_code)
return render_unprocessable_entity field: :title unless tag_name
return render_unprocessable_entity field: :title if tag_name.name.include?(':')
base_revision_id = params[:base_revision_id].presence
ApplicationRecord.transaction do
page.lock!
old_title = page.title
tag = page.tag_name.tag
if tag && title != old_title
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
tag.lock!
old_title = tag_name.name
if title != old_title
TagVersioning.ensure_snapshot!(tag, created_by_user: current_user)
end
page.tag_name.update!(name: title) if title != old_title
tag_name.update!(name: title) if title != old_title
message = params[:message].presence
Wiki::Commit.content!(page:,
body:,
created_user: current_user,
message:,
base_revision_id:)
TagDescriptionOperation.update!(locale, tag:, body:, updated_by_user: current_user,
message:, base_revision_id:)
if tag && title != old_title
if title != old_title
tag.reload
TagVersionRecorder.record!(tag:, event_type: :update, created_by_user: current_user)
end
end
head :ok
rescue ActiveRecord::RecordInvalid => e
render_validation_error e.record
rescue ActiveRecord::RecordNotUnique
render_record_not_unique
rescue TagGroupMarkdownParser::InvalidDocument => error
render_unprocessable_entity(error.message)
end
def changes
locale = resolve_locale!
id = params[:id].presence
description = id && TagDescription.find_by(id:)
description = id && TagDescription.find_by(id:, language_code: locale.language_code)
if description
return render json: description_changes(TagDescription.where(id: description.id))
return render json: description_changes(locale, TagDescription.where(id: description.id))
end
pages = id ? WikiPage.where(id:) : unmigrated_wiki_pages
revisions = WikiRevision.joins(wiki_page: :tag_name)
.where(wiki_page_id: pages.select(:id))
.includes(:created_user, wiki_page: { tag_name: :tag })
.order(id: :desc).limit(200)
json = revisions.map { |rev|
{ revision_id: rev.id,
pred: rev.base_revision_id,
succ: nil,
wiki_page: { id: rev.wiki_page_id,
title: rev.wiki_page.title,
deprecated_at: rev.wiki_page.deprecated_at },
user: rev.created_user && { id: rev.created_user.id, name: rev.created_user.name },
kind: rev.kind,
message: rev.message,
timestamp: rev.created_at }
}
tag_group = id && TagGroup.find_by(id:)
if tag_group
return render json: tag_group_changes(TagGroupVersion.where(tag_group_id: tag_group.id))
end
json = []
unless id
json.concat(description_changes(TagDescription.where(language_code: 'ja')))
json.sort_by! { |row| -row[:revision_id] }
json.concat(tag_group_changes(TagGroupVersion.all))
json.concat(description_changes(locale,
TagDescription.where(language_code: locale.language_code)))
json = json.each_with_index.sort_by do |row, index|
[-row.fetch(:timestamp).to_f, index]
end.map(&:first)
end
render json: json.first(200)
end
private
def render_wiki_page_or_404 page
return head :not_found unless page
rev = find_revision(page)
return head :not_found unless rev
if rev.redirect?
return (
redirect_to wiki_page_by_title_path(title: rev.redirect_page.title),
status: :moved_permanently)
def render_tag_group locale, group
version = if params[:version].present?
group_version(group, params[:version])
else
current_group_version(group)
end
return head :not_found unless version
body = rev.body
revision_id = rev.id
pred = page.pred_revision_id(revision_id)
succ = page.succ_revision_id(revision_id)
updated_at = rev.created_at
render json: WikiPageRepr.base(page).merge(body:, revision_id:, pred:, succ:, updated_at:)
representation =
if params[:version].present?
TagGroupRepr.historical(locale, version)
else
TagGroupRepr.one(locale, group)
end
render_group_representation(representation, version)
end
def render_wiki_page_by_tag_description description
def render_group_version locale, version
representation = TagGroupRepr.historical(locale, version)
TagGroupMarkdownRenderer.render(representation)
end
def render_group_representation representation, version, body: nil, status: :ok
body ||= TagGroupMarkdownRenderer.render(representation)
versions = TagGroupVersion.where(tag_group_id: version.tag_group_id)
pred = versions.find_by(version_no: version.version_no - 1)&.id
succ = versions.find_by(version_no: version.version_no + 1)&.id
updated_at = version.created_at
render json: {
'id' => version.tag_group_id,
'resource_type' => 'tag_group',
'title' => "#{ TagGroupRepr::PREFIX }#{ representation.fetch('name') }",
'deprecated_at' => nil,
'body' => body,
'revision_id' => version.id,
'pred' => pred,
'succ' => succ,
'updated_at' => updated_at }, status:
rescue TagGroupMarkdownRenderer::UnsupportedStructure => error
render_unprocessable_entity(error.message)
end
def current_group_version group
TagGroupVersion.find_by(tag_group_id: group.id, version_no: group.version_no)
end
def group_version group, version_id
TagGroupVersion.find_by(tag_group_id: group.id, id: version_id)
end
def wiki_diff from_body, to_body
diffs = Diff::LCS.sdiff(from_body.to_s.lines, to_body.to_s.lines)
diffs.map { |change|
case change.action
when ?=
{ type: 'context', content: change.old_element }
when ?!
[{ type: 'removed', content: change.old_element },
{ type: 'added', content: change.new_element }]
when ?+
{ type: 'added', content: change.new_element }
when ?-
{ type: 'removed', content: change.old_element }
end
}.flatten.compact
end
def tag_group_changes versions
rows = versions.includes(:created_by_user).order(id: :desc).limit(200).to_a
groups = TagGroup.where(id: rows.map(&:tag_group_id).uniq).index_by(&:id)
predecessors = TagGroupVersion.where(tag_group_id: groups.keys,
version_no: rows.map { _1.version_no - 1 })
.pluck(:tag_group_id, :version_no, :id)
.to_h { |group_id, number, version_id| [[group_id, number], version_id] }
rows.map do |version|
group = groups.fetch(version.tag_group_id)
user = version.created_by_user
{ revision_id: version.id,
pred: predecessors[[group.id, version.version_no - 1]],
succ: nil,
wiki_page: { id: group.id, title: "#{ TagGroupRepr::PREFIX }#{ version.name }",
deprecated_at: nil },
user: user && { id: user.id, name: user.name },
kind: 'content', message: nil, timestamp: version.created_at }
end
end
def render_wiki_page_by_tag_description locale, description
versions = description_versions(description)
historical = params[:version].present?
rev = historical ? versions.find_by(id: params[:version]) :
@@ -255,7 +401,7 @@ class WikiPagesController < ApplicationController
succ = versions.find_by(version_no: rev.version_no + 1)&.id
body = historical ? rev.body : description.body
updated_at = historical ? rev.created_at : description.updated_at
render json: WikiPageRepr.tag_description(description).merge(
render json: WikiPageRepr.tag_description(locale, description).merge(
body:, revision_id: rev.id, pred:, succ:, updated_at:)
end
@@ -264,20 +410,14 @@ class WikiPagesController < ApplicationController
language_code: description.language_code)
end
def tag_description_by_title title
def tag_description_by_title locale, title
TagDescription.joins(tag: :tag_names)
.find_by(language_code: 'ja', tag_names: { language_code: 'ja', name: title })
.find_by(language_code: locale.language_code,
tag_names: { language_code: locale.language_code, name: title })
end
def unmigrated_wiki_pages
descriptions = TagDescription.where(language_code: 'ja')
WikiPage.joins(:tag_name).where.not(id: descriptions.select(:id))
.where('tag_names.tag_id IS NULL OR tag_names.tag_id NOT IN (?)',
descriptions.select(:tag_id))
end
def description_changes descriptions
versions = TagDescriptionVersion.where(language_code: 'ja',
def description_changes locale, descriptions
versions = TagDescriptionVersion.where(language_code: locale.language_code,
tag_id: descriptions.select(:tag_id))
.order(id: :desc).limit(200).to_a
return [] if versions.empty?
@@ -286,7 +426,7 @@ class WikiPagesController < ApplicationController
.includes(tag: :tag_names).index_by(&:tag_id)
users = User.where(id: versions.map(&:created_by_user_id)).index_by(&:id)
predecessors = TagDescriptionVersion.where(
language_code: 'ja', tag_id: by_tag.keys,
language_code: locale.language_code, tag_id: by_tag.keys,
version_no: versions.map { |version| version.version_no - 1 })
.pluck(:tag_id, :version_no, :id)
.to_h { |tag_id, version_no, id| [[tag_id, version_no], id] }
@@ -296,7 +436,7 @@ class WikiPagesController < ApplicationController
{ revision_id: version.id,
pred: predecessors[[version.tag_id, version.version_no - 1]],
succ: nil,
wiki_page: { id: description.id, title: description.tag.name('ja'),
wiki_page: { id: description.id, title: description.tag.name(locale.language_code),
deprecated_at: description.tag.deprecated_at },
user: user && { id: user.id, name: user.name },
kind: 'content',
@@ -305,14 +445,6 @@ class WikiPagesController < ApplicationController
end
end
def find_revision page
if params[:version].present?
page.wiki_revisions.find_by(id: params[:version])
else
page.current_revision
end
end
def render_wiki_conflict err
render json: { error: 'conflict', message: err.message }, status: :conflict
end
+2
ファイルの表示
@@ -1,4 +1,6 @@
class ExternalTag < ApplicationRecord
before_create { SharedResourceIdAllocator.assign!(self) }
enum :platform, nico: 'nico'
validates :platform, presence: true, inclusion: { in: ExternalTag.platforms.keys }
+1 -1
ファイルの表示
@@ -1,5 +1,5 @@
class Language < ApplicationRecord
has_many :locales, foreign_key: :language_code, primary_key: :code
enum :code, ja: 'ja'
enum :code, ja: 'ja', ain: 'ain'
end
+25 -14
ファイルの表示
@@ -1,27 +1,38 @@
class Locale < ApplicationRecord
after_create_commit :generate_tag_names!
attr_accessor :certified_tag_names_path
around_save :save_with_tag_names!, if: :will_generate_tag_names?
belongs_to :language, foreign_key: :language_code, primary_key: :code
belongs_to :script, foreign_key: :script_code, primary_key: :code
enum :code, ja: 'ja', en: 'en', fr: 'fr'
# Enum deserialisation uses the first key for each stored value.
# Keep the legacy key for existing predicates, scopes and query arguments.
enum :code, { ja: 'ja', 'ain-Kana' => 'ain-Kana', ain_kana: 'ain-Kana' }
def code= value
super(self.class.codes.fetch(value.to_s, value))
end
def self.nipponese = Locale.find('ja')
private
def generate_tag_names!
ApplicationRecord.transaction do
tag_ids = TagName.where(language_code:, primary_flg: true).pluck(:tag_id)
Tag.where.not(id: tag_ids).find_each do
TagName.create!(tag_id: _1.id,
language_code:,
name: TagName.generate_name(self, _1, _1.tag_name('ja')),
script_code:,
primary_flg: true,
auto_generated: true,
canonical_id: nil)
end
def will_generate_tag_names?
deprecated_at.nil? && (will_save_change_to_code? || will_save_change_to_deprecated_at?)
end
def save_with_tag_names!
completed = self.class.transaction(requires_new: true) do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
saved = yield
raise ActiveRecord::RecordNotSaved, 'Locale save was halted' unless saved
TagNameCompletion.for_locale!(self)
true
end
unless completed
raise ActiveRecord::RecordNotSaved, 'Locale tag completion was rolled back'
end
end
end
+5
ファイルの表示
@@ -1,6 +1,11 @@
class PostVersion < ApplicationRecord
include VersionRecord
scope :with_previous_version, -> {
joins('LEFT JOIN post_versions prev ON prev.post_id = post_versions.post_id ' +
'AND prev.version_no = post_versions.version_no - 1')
}
belongs_to :post
belongs_to :parent, class_name: 'Post', optional: true
+7 -1
ファイルの表示
@@ -1,5 +1,11 @@
class Script < ApplicationRecord
has_many :locales, foreign_key: :script_code, primary_key: :code
enum :code, Jpan: 'Jpan'
# Canonical keys come first so reads and plucks retain ISO 15924 spelling.
# Legacy keys preserve the existing enum predicates, scopes and queries.
enum :code, { 'Jpan' => 'Jpan', 'Kana' => 'Kana', jpan: 'Jpan', kana: 'Kana' }
def code= value
super(self.class.codes.fetch(value.to_s, value))
end
end
+39 -49
ファイルの表示
@@ -2,6 +2,8 @@ require 'set'
class Tag < ApplicationRecord
before_create { SharedResourceIdAllocator.assign!(self) }
class NicoTagNormalisationError < ArgumentError
;
end
@@ -51,6 +53,7 @@ class Tag < ApplicationRecord
has_many :tag_versions
has_many :tag_names
has_many :tag_descriptions
attribute :version_no, :integer, default: 1
@@ -88,7 +91,6 @@ class Tag < ApplicationRecord
end
end
def name(language_code) = tag_name(language_code)&.name
def wiki_page = tag_name('ja')&.wiki_page
def system?
tag_name('ja')&.name.in?(SYSTEM_TAG_NAMES)
@@ -96,13 +98,24 @@ class Tag < ApplicationRecord
def deprecated? = deprecated_at?
def has_wiki = wiki_page.present?
def has_description? language_code:
if tag_descriptions.loaded?
tag_descriptions.any? { _1.language_code == language_code }
else
tag_descriptions.exists?(language_code:)
end
end
def material_id = materials.first&.id
def has_deerjikists = deerjikists.loaded? ? deerjikists.any? : deerjikists.exists?
def snapshot_aliases(language_code)
def snapshot_aliases language_code
if tag_names.loaded?
return tag_names.select { _1.language_code == language_code && !_1.primary? }
.sort_by(&:name).map(&:name)
end
tag_names.where(language_code:, primary_flg: false).order(:name).pluck(:name)
end
@@ -129,7 +142,8 @@ class Tag < ApplicationRecord
with_tagme: true,
with_no_deerjikist: true,
deny_deprecated: false,
with_sections: false
with_sections: false,
created_by_user: nil
if tag_names.any? { |n| n.downcase.start_with?('nico:') }
raise NicoTagNormalisationError
end
@@ -158,7 +172,8 @@ class Tag < ApplicationRecord
name = TagName.canonicalise(locale, name).first
find_or_create_by_tag_name!(locale, name, category: (cat || :general)).tap do |tag|
find_or_create_by_tag_name!(locale, name, category: (cat || :general),
created_by_user:).tap do |tag|
if deny_deprecated && tag.deprecated?
raise DeprecatedTagNormalisationError, [tag.name(locale.language_code)]
end
@@ -246,28 +261,28 @@ class Tag < ApplicationRecord
[left_end_ms, right_end_ms].max
end
def find_or_create_by_tag_name! locale, name, category:
def find_or_create_by_tag_name! locale, name, category:, created_by_user: nil
language_code = locale.language_code
name = name.to_s.strip
ApplicationRecord.transaction do
tn = TagName.find_or_create_by!(language_code:, name:) do
_1.script_code = locale.script_code
_1.primary_flg = true
end
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
tn = TagName.lock.find_by(language_code:, name:)
next tn.tag if tn
tag = tn.tag
next tag if tag
tag = Tag.create!(tag_name_id: tn.id, category:)
tn.update!(tag:, primary_flg: true, canonical_id: nil, auto_generated: false)
generate_tag_names!(locale, tag, tn)
tag = Tag.new(category:)
tag.tag_names.build(language_code:, name:, script_code: locale.script_code,
primary_flg: true, auto_generated: false)
TagNameCompletion.for_tag!(tag, created_by_user:)
tag.association(:tag_names).reset
tag
end
rescue ActiveRecord::RecordNotUnique
retry
existing = TagName.find_by(language_code:, name:)
raise unless existing
existing.tag
end
def merge_tags! target_tag, source_tags, created_by_user: nil
@@ -294,20 +309,18 @@ class Tag < ApplicationRecord
end
end
source_tag_name = source_tag.tag_name('ja')
if source_tag_name.wiki_page.present?
raise ActiveRecord::RecordInvalid.new(source_tag_name)
if source_tag.tag_descriptions.exists?
source_tag.errors.add(:base, :invalid)
raise ActiveRecord::RecordInvalid, source_tag
end
TagVersioning.record!(source_tag, event_type: :discard, created_by_user:)
source_tag.tag_names.update_all(tag_id: target_tag.id, primary_flg: false,
updated_at: Time.current)
target_languages = target_tag.tag_names.where(primary_flg: true).pluck(:language_code)
source_tag.tag_names.where(language_code: target_languages)
.update_all(primary_flg: false, auto_generated: false, updated_at: Time.current)
source_tag.tag_names.update_all(tag_id: target_tag.id, updated_at: Time.current)
source_tag.destroy!
source_tag_name.update_columns(canonical_id: target_tag.tag_name_id,
updated_at: Time.current)
TagVersioning.record!(target_tag, event_type: :update, created_by_user:)
end
@@ -349,29 +362,6 @@ class Tag < ApplicationRecord
total_s * 1_000 + match[:ms].to_s.ljust(3, '0')[0, 3].to_i
end
private
def generate_tag_names! locale, tag, base_tag_name
ApplicationRecord.transaction do
locales =
Locale
.where(deprecated_at: nil)
.where.not(language_code: locale.language_code)
.order(:language_code, :created_at, :script_code)
.uniq { _1.language_code }
locales.each do |locale|
language_code = locale.language_code
script_code = locale.script_code
TagName.create!(tag_id: tag.id,
language_code:,
name: TagName.generate_name(locale, tag, base_tag_name),
script_code:,
primary_flg: true,
auto_generated: true,
canonical_id: nil)
end
end
end
end
private
+2
ファイルの表示
@@ -1,3 +1,5 @@
class TagDescription < ApplicationRecord
before_create { SharedResourceIdAllocator.assign!(self) }
belongs_to :tag
end
+1 -1
ファイルの表示
@@ -1,3 +1,3 @@
class TagDescriptionVersion < ApplicationRecord
;
before_create { SharedResourceIdAllocator.assign!(self) }
end
+27
ファイルの表示
@@ -0,0 +1,27 @@
class TagGroup < ApplicationRecord
before_create { SharedResourceIdAllocator.assign!(self) }
before_destroy :delete_current_structure!
belongs_to :created_by_user, class_name: 'User', optional: true
belongs_to :updated_by_user, class_name: 'User', optional: true
has_many :tag_group_tags, dependent: :delete_all
has_many :tags, through: :tag_group_tags
has_many :tag_group_groups, dependent: :delete_all
has_many :child_tag_groups, through: :tag_group_groups
has_many :placements, -> { order(:position) },
class_name: 'TagGroupPlacement', dependent: :delete_all
has_many :versions, -> { order(:version_no) },
class_name: 'TagGroupVersion'
validates :name, presence: true, uniqueness: true, length: { maximum: 255 }
validates :version_no, numericality: { only_integer: true, greater_than: 0 }
private
def delete_current_structure!
placements.reorder(position: :desc).each(&:delete)
tag_group_tags.delete_all
tag_group_groups.delete_all
end
end
+7
ファイルの表示
@@ -0,0 +1,7 @@
class TagGroupGroup < ApplicationRecord
self.primary_key = [:tag_group_id, :child_tag_group_id]
belongs_to :tag_group
belongs_to :child_tag_group, class_name: 'TagGroup'
belongs_to :created_by_user, class_name: 'User', optional: true
end
+9
ファイルの表示
@@ -0,0 +1,9 @@
class TagGroupPlacement < ApplicationRecord
self.primary_key = [:tag_group_id, :position]
belongs_to :tag_group
belongs_to :tag, optional: true
belongs_to :child_tag_group, class_name: 'TagGroup', optional: true
belongs_to :created_by_user, class_name: 'User', optional: true
belongs_to :updated_by_user, class_name: 'User', optional: true
end
+7
ファイルの表示
@@ -0,0 +1,7 @@
class TagGroupTag < ApplicationRecord
self.primary_key = [:tag_group_id, :tag_id]
belongs_to :tag_group
belongs_to :tag
belongs_to :created_by_user, class_name: 'User', optional: true
end
+9
ファイルの表示
@@ -0,0 +1,9 @@
class TagGroupVersion < ApplicationRecord
include VersionRecord
before_create { SharedResourceIdAllocator.assign!(self) }
belongs_to :tag_group
scope :ordered, -> { order(:version_no) }
end
+25 -16
ファイルの表示
@@ -1,14 +1,12 @@
class TagName < ApplicationRecord
belongs_to :language, foreign_key: :language_code, primary_key: :code
belongs_to :tag, optional: true
has_one :wiki_page
belongs_to :tag
validates :name, presence: true,
length: { maximum: 255 },
uniqueness: { scope: :language_code }
validate :alias_name_must_not_have_prefix
validate :alias_must_not_have_wiki_page
validate :name_must_be_sanitised
validate :name_mustnt_start_with_nico
validate :generated_name_must_be_reserved
@@ -21,22 +19,37 @@ class TagName < ApplicationRecord
tns = TagName.where(language_code: locale.language_code, name: names).index_by(&:name)
primary_names = TagName.where(
language_code: locale.language_code, tag_id: tns.values.map(&:tag_id),
primary_flg: true).pluck(:tag_id, :name).to_h
names.map { |name|
tag_name = tns[name]
if !tag_name || tag_name.primary? || tag_name.tag_id.nil?
if !tag_name || tag_name.primary?
name
else
TagName.find_by(language_code: locale.language_code,
tag_id: tag_name.tag_id,
primary_flg: true)&.name || name
primary_names[tag_name.tag_id] || name
end
}.uniq
end
def self.generate_name locale, tag, base_tag_name
# TODO: 言語ごとの自動命名ロジック完成したら書く.
"Tag_##{ tag.id }"
fallback = "Tag_##{ tag.id }"
name =
case [locale.language_code, base_tag_name&.language_code]
when ['ain', 'ja']
TagNameTranscription.ainu(base_tag_name.name)
when ['ja', 'ain']
base_tag_name.name
else
return fallback
end
candidate = new(tag:, name:, language_code: locale.language_code,
script_code: locale.script_code,
primary_flg: true, auto_generated: true)
candidate.valid? ? name : fallback
rescue TagNameTranscription::Unconvertible
fallback
end
private
@@ -53,13 +66,9 @@ class TagName < ApplicationRecord
end
end
def alias_must_not_have_wiki_page
if !(primary?) && wiki_page
errors.add :primary_flg, 'Wiki 参照がある名前はエーリアスになれません.'
end
end
def name_must_be_sanitised
return if auto_generated && name == "Tag_##{ tag_id }"
if name? && name != TagNameSanitisationRule.sanitise(name)
errors.add :name, '名前に使用できない文字が含まれてゐます.'
end
+25 -26
ファイルの表示
@@ -16,37 +16,36 @@ class TagNameSanitisationRule < ApplicationRecord
def apply!
Language.find_each do |language|
TagName.where(language:).find_each do |tn|
name = sanitise(tn.name)
next if name == tn.name
TagName.where(language:).find_each do |candidate|
TagName.transaction do
tn = TagName.find_by(id: candidate.id)
next unless tn
name = sanitise(tn.name)
next if name == tn.name
existing_tn = TagName.find_by(language:, name:)
if existing_tn
unless existing_tn.primary_flg
existing_tn = TagName.find_by!(language:,
tag_id: existing_tn.tag_id,
primary_flg: true)
source_tag = tn.tag
target_tag = existing_tn ? existing_tn.tag : source_tag
[source_tag, target_tag].uniq(&:id).sort_by(&:id).each(&:lock!)
TagVersioning.ensure_snapshot!(source_tag, created_by_user: nil)
if existing_tn && target_tag != source_tag
Tag.merge_tags!(target_tag, source_tag)
tn.reload.destroy!
elsif existing_tn
if tn.primary?
existing_tn.destroy!
tn.update_columns(name:, updated_at: Time.current)
else
tn.destroy!
end
next if existing_tn.id == tn.id
existing_tag = existing_tn.tag
source_tag = tn.tag
if existing_tag
Tag.merge_tags!(existing_tag, source_tag) if tn.tag
elsif source_tag
tn.update_columns(tag_id: nil, updated_at: Time.current)
source_tag.update_columns(tag_name_id: existing_tn.id, updated_at: Time.current)
existing_tn.update_columns(tag_id: source_tag.id, updated_at: Time.current)
end
tn.destroy!
next
else
# TagName 側の自動サニタイズを回避
tn.update_columns(name:, updated_at: Time.current)
end
# TagName 側の自動サニタイズを回避
tn.update_columns(name:, updated_at: Time.current)
TagVersioning.record!(target_tag, event_type: :update, created_by_user: nil)
end
end
end
+4 -4
ファイルの表示
@@ -13,10 +13,10 @@ class User < ApplicationRecord
has_many :ip_addresses, through: :user_ips
has_many :user_post_views, dependent: :destroy
has_many :viewed_posts, through: :user_post_views, source: :post
has_many :created_wiki_pages,
class_name: 'WikiPage', foreign_key: :created_user_id, dependent: :nullify
has_many :updated_wiki_pages,
class_name: 'WikiPage', foreign_key: :updated_user_id, dependent: :nullify
has_many :created_tag_descriptions, class_name: 'TagDescription',
foreign_key: :created_by_user_id, dependent: :nullify
has_many :updated_tag_descriptions, class_name: 'TagDescription',
foreign_key: :updated_by_user_id, dependent: :nullify
def viewed?(post) = user_post_views.exists?(post_id: post.id)
-15
ファイルの表示
@@ -1,15 +0,0 @@
class WikiLine < ApplicationRecord
has_many :wiki_revision_lines, dependent: :restrict_with_exception
validates :sha256, presence: true, uniqueness: true, length: { is: 64 }
validates :body, presence: true
def self.upsert_by_body! body
sha = Digest::SHA256.hexdigest(body)
now = Time.current
upsert(sha256: sha, body:, created_at: now, updated_at: now)
find_by!(sha256: sha)
end
end
-58
ファイルの表示
@@ -1,58 +0,0 @@
require 'set'
class WikiPage < ApplicationRecord
include MyDiscard
has_many :wiki_revisions, dependent: :destroy
belongs_to :created_user, class_name: 'User'
belongs_to :updated_user, class_name: 'User'
has_many :redirected_from_revisions,
class_name: 'WikiRevision',
foreign_key: :redirect_page_id,
dependent: :nullify
has_many :wiki_versions
attribute :version_no, :integer, default: 1
belongs_to :tag_name
validates :tag_name, presence: true
validates :body, presence: true
def title = tag_name.name
def deprecated_at = tag_name.tag&.deprecated_at
def title= val
(self.tag_name ||= build_tag_name).name = val
end
def current_revision = wiki_revisions.order(id: :desc).first
def resolve_redirect limit: 10
page = self
visited = Set.new
limit.times do
return page if visited.include?(page.id)
visited.add(page.id)
rev = page.current_revision
return page if !(rev&.redirect?) || !(rev.redirect_page)
page = rev.redirect_page
end
page
end
def pred_revision_id revision_id
wiki_revisions.where('id < ?', revision_id).order(id: :desc).limit(1).pick(:id)
end
def succ_revision_id revision_id
wiki_revisions.where('id > ?', revision_id).order(id: :asc).limit(1).pick(:id)
end
end
-55
ファイルの表示
@@ -1,55 +0,0 @@
class WikiRevision < ApplicationRecord
belongs_to :wiki_page
belongs_to :base_revision, class_name: 'WikiRevision', optional: true
belongs_to :created_user, class_name: 'User'
belongs_to :redirect_page, class_name: 'WikiPage', optional: true
has_many :wiki_revision_lines, dependent: :delete_all
has_many :wiki_lines, through: :wiki_revision_lines
enum :kind, content: 0, redirect: 1
validates :kind, presence: true
validates :lines_count, numericality: { only_integer: true, greater_than_or_equal_to: 0 }
validates :tree_sha256, length: { is: 64 }, allow_nil: true
validate :kind_consistency
def body
return unless content?
wiki_revision_lines
.includes(:wiki_line)
.order(:position)
.map { |rev| rev.wiki_line.body }
.join("\n")
end
private
def kind_consistency
if content?
if tree_sha256.blank?
errors.add(:tree_sha256, '種類がページの場合は必須です.')
end
if redirect_page_id.present?
errors.add(:redirect_page_id, '種類がページの場合は空である必要があります.')
end
end
if redirect?
if redirect_page_id.blank?
errors.add(:redirect_page_id, '種類がリダイレクトの場合は必須です.')
end
if tree_sha256.present?
errors.add(:tree_sha256, '種類がリダイレクトの場合は空である必要があります.')
end
if lines_count.to_i > 0
errors.add(:lines_count, '種類がリダイレクトの場合は 0 である必要があります.')
end
end
end
end
-8
ファイルの表示
@@ -1,8 +0,0 @@
class WikiRevisionLine < ApplicationRecord
belongs_to :wiki_revision
belongs_to :wiki_line
validates :position, presence: true,
numericality: { only_integer: true, greater_than_or_equal_to: 0 }
validates :position, uniqueness: { scope: :wiki_revision_id }
end
-8
ファイルの表示
@@ -1,8 +0,0 @@
class WikiVersion < ApplicationRecord
include VersionRecord
belongs_to :wiki_page
validates :title, presence: true
validates :body, presence: true
end
+103
ファイルの表示
@@ -0,0 +1,103 @@
# Counts uses recorded values only; history/diff endpoints retain their own contract.
class CountsVersionRepr
def self.many(rows, resource:, language_code:, at: nil, inclusive: true)
new(rows.to_a, resource:, language_code:, at:, inclusive:).serialise
end
def initialize(rows, resource:, language_code:, at:, inclusive:)
@rows = rows
@at = at
@inclusive = inclusive
@resource = resource
@language = language_code
@users = User.where(id: rows.map(&:created_by_user_id).compact).pluck(:id, :name).to_h
@languages = master_names(Language)
@scripts = master_names(Script)
@tag_versions = {}
load_tag_versions if resource == 'posts'
end
def serialise
@rows.map do |row|
resource_id = row.public_send(@resource == 'posts' ? :post_id : :tag_id)
common = { id: row.id, resource_id:,
version_no: row.version_no, is_latest: row.attributes['counts_latest'].to_i == 1,
created_at: row.created_at.iso8601,
created_by_user: row.created_by_user_id &&
{ id: row.created_by_user_id, name: @users[row.created_by_user_id] } }
common.merge(@resource == 'posts' ? post(row) : tag(row))
end
end
private
def master_names(model)
model.select('code AS counts_code', :name).to_h do |record|
[record.attributes['counts_code'], record.name]
end
end
def json(value)
value.is_a?(String) ? JSON.parse(value) : value
end
def load_tag_versions
pairs = @rows.flat_map { Array(json(_1.tags_json)) }
.filter_map { |entry| [entry['tag_id'], entry['version_no']] if entry['tag_id'] }
.uniq
pairs.each_slice(500) do |batch|
placeholders = batch.map { '(?, ?)' }.join(', ')
scope = TagVersion.all
if @at
scope = scope.where("created_at #{ @inclusive ? '<=' : '<' } ?", @at)
end
scope.where(["(tag_id, version_no) IN (#{ placeholders })", *batch.flatten])
.select(:tag_id, :version_no, :name, :category, :names).each do |version|
@tag_versions[[version.tag_id, version.version_no]] = version
end
end
end
def names(value)
(json(value) || {}).sort.map do |language, entry|
aliases = entry.fetch('aliases', []).map do |alias_entry|
alias_entry.merge('script_name' => @scripts[alias_entry['script_code']])
end
{ language_code: language, language_name: @languages[language],
script_code: entry['script_code'], script_name: @scripts[entry['script_code']],
name: entry['name'], auto_generated: entry['auto_generated'], aliases: }
end
end
def post(row)
snapshots = Array(json(row.tags_json)).map do |entry|
version = @tag_versions[[entry['tag_id'], entry['version_no']]]
{ id: entry['tag_id'] || entry['external_tag_id'],
kind: entry['tag_id'] ? 'internal' : 'external',
tag_id: entry['tag_id'], external_tag_id: entry['external_tag_id'],
version_no: entry['version_no'], name: entry['name'] || version&.name,
category: entry['category'] || version&.category,
sections: entry.fetch('sections', []), names: version ? names(version.names) : [] }
end
{ title: row.title, url: row.url, thumbnail_base: row.thumbnail_base,
tags: snapshots, parents: row.parent_post_ids.to_s.split.map { { id: _1.to_i } },
video_ms: row.video_ms,
original_created_from: row.original_created_from&.iso8601,
original_created_before: row.original_created_before&.iso8601 }
end
def tag(row)
recorded_names = names(row.names)
selected = recorded_names.find { _1[:language_code] == @language }
aliases = recorded_names.flat_map do |entry|
entry[:aliases].map {
_1.merge('language_code' => entry[:language_code],
'language_name' => entry[:language_name])
}
end
{ name: selected&.fetch(:name) || row.name, names: recorded_names,
category: row.category, aliases:,
parents: row.parent_tag_ids.to_s.split.map { { id: _1.to_i } },
deprecated: row.deprecated_at&.iso8601 }
end
end
+205
ファイルの表示
@@ -0,0 +1,205 @@
module TagGroupRepr
PREFIX = 'タググループ:'
module_function
def name_from_title title
prefix = [PREFIX, 'tag_group:'].find { title.downcase.start_with?(_1) }
title[prefix.length..] if prefix
end
def many locale, groups
ids = groups.map(&:id)
return [] if ids.empty?
tag_members = TagGroupTag.where(tag_group_id: ids)
.group_by(&:tag_group_id)
group_members = TagGroupGroup.where(tag_group_id: ids)
.group_by(&:tag_group_id)
placements = TagGroupPlacement.where(tag_group_id: ids).order(:position)
.group_by(&:tag_group_id)
target_ids =
tag_members.values.flatten.map(&:tag_id) +
placements.values.flatten.filter_map(&:tag_id)
child_ids =
group_members.values.flatten.map(&:child_tag_group_id) +
placements.values.flatten.filter_map(&:child_tag_group_id)
tags = Tag.select(:id, :category).where(id: target_ids.uniq).index_by(&:id)
names =
TagName
.where(tag_id: target_ids.uniq,
language_code: locale.language_code,
primary_flg: true)
.pluck(:tag_id, :name)
.to_h
child_groups = TagGroup.where(id: child_ids.uniq).index_by(&:id)
groups.map do |group|
group_json(group,
tag_members.fetch(group.id, []),
group_members.fetch(group.id, []),
placements.fetch(group.id, []), tags, names, child_groups)
end
end
def one locale, group
many(locale, [group]).first
end
def tag_group wiki_page
{ 'id' => wiki_page.id,
'title' => "#{ PREFIX }#{ wiki_page.name }",
'deprecated_at' => nil,
'created_at' => wiki_page.created_at,
'updated_at' => wiki_page.updated_at }
end
def current_version group
TagGroupVersion.find_by(tag_group_id: group.id, version_no: group.version_no)
end
def historical locale, version
members = Array(version.members)
placements = Array(version.placements)
tag_references = members.filter_map { |member| member['tag_id'] } +
collect_ids(placements, 'tag_id')
group_references = members.filter_map { |member| member['tag_group_id'] } +
collect_ids(placements, 'tag_group_id')
tag_versions = historical_tag_versions(tag_references)
group_versions = historical_group_versions(group_references)
tag_version_numbers = members.filter_map do |member|
next unless member['tag_id']
[member.fetch('tag_id'), member.fetch('version_no')]
end.to_h
group_version_numbers = members.filter_map do |member|
next unless member['tag_group_id']
[member.fetch('tag_group_id'), member.fetch('version_no')]
end.to_h
historical_members = members.map do |member|
historical_member(locale, member, tag_versions, group_versions)
end
historical_placements = placements.map do |node|
historical_placement(locale, node, tag_versions, group_versions,
tag_version_numbers, group_version_numbers)
end
{ 'id' => version.tag_group_id,
'name' => version.name,
'description' => version.description,
'version_no' => version.version_no,
'members' => historical_members,
'placements' => historical_placements,
'created_at' => version.created_at,
'updated_at' => version.created_at,
'created_by_user' => user_json(version.created_by_user),
'updated_by_user' => user_json(version.created_by_user) }
end
def group_json group, tag_members, group_members, rows, tags, names, child_groups
tree = placement_tree(rows, tags, names, child_groups)
members = tag_members.sort_by(&:tag_id).map do |member|
{ 'tag' => tag_json(tags.fetch(member.tag_id), names[member.tag_id]) }
end
members.concat(group_members.sort_by(&:child_tag_group_id).map do |member|
{ 'tag_group' => group_json_reference(child_groups.fetch(member.child_tag_group_id)) }
end)
{ 'id' => group.id,
'name' => group.name,
'description' => group.description,
'version_no' => group.version_no,
'members' => members,
'placements' => tree,
'created_at' => group.created_at,
'updated_at' => group.updated_at,
'created_by_user' => user_json(group.created_by_user),
'updated_by_user' => user_json(group.updated_by_user) }
end
def user_json user
UserRepr.base(user) if user
end
def placement_tree rows, tags, names, child_groups
by_parent = rows.group_by(&:parent_position)
build = lambda do |parent|
by_parent.fetch(parent, []).sort_by(&:position).map do |row|
node = if row.tag_id
{ 'tag' => tag_json(tags.fetch(row.tag_id), names[row.tag_id]) }
elsif row.child_tag_group_id
{ 'tag_group' => group_json_reference(child_groups.fetch(row.child_tag_group_id)) }
else
{ 'section_name' => row.section_name, 'description' => row.description }
end
node.merge('children' => build.call(row.position))
end
end
build.call(nil)
end
def tag_json tag, name
{ 'id' => tag.id, 'name' => name, 'category' => tag.category }
end
def group_json_reference group
{ 'id' => group.id, 'name' => group.name }
end
def collect_ids nodes, key
nodes.flat_map do |node|
[node[key], *collect_ids(Array(node['children']), key)].compact
end
end
def historical_tag_versions ids
TagVersion.where(tag_id: ids.uniq).to_a.index_by do |version|
[version.tag_id, version.version_no]
end
end
def historical_group_versions ids
TagGroupVersion.where(tag_group_id: ids.uniq).to_a.index_by do |version|
[version.tag_group_id, version.version_no]
end
end
def historical_member locale, member, tag_versions, group_versions
if member['tag_id']
id = member.fetch('tag_id')
version = tag_versions.fetch([id, member.fetch('version_no')])
{ 'tag' => { 'id' => id, 'name' => version_name(locale, version),
'category' => version.category } }
else
id = member.fetch('tag_group_id')
version = group_versions.fetch([id, member.fetch('version_no')])
{ 'tag_group' => { 'id' => id, 'name' => version.name } }
end
end
def historical_placement locale, node, tag_versions, group_versions,
tag_version_numbers, group_version_numbers
result = if node['tag_id']
id = node.fetch('tag_id')
version_no = tag_version_numbers.fetch(id)
version = tag_versions.fetch([id, version_no])
{ 'tag' => { 'id' => id, 'name' => version_name(locale, version),
'category' => version.category } }
elsif node['tag_group_id']
id = node.fetch('tag_group_id')
version_no = group_version_numbers.fetch(id)
version = group_versions.fetch([id, version_no])
{ 'tag_group' => { 'id' => id, 'name' => version.name } }
else
{ 'section_name' => node.fetch('section_name'), 'description' => node['description'] }
end
result.merge('children' => Array(node['children']).map do |child|
historical_placement(locale, child, tag_versions, group_versions,
tag_version_numbers, group_version_numbers)
end)
end
def version_name locale, version
version.names[locale.language_code]&.fetch('name')
end
end
+11 -4
ファイルの表示
@@ -3,21 +3,28 @@
module TagRepr
BASE = { only: [:id, :category, :post_count, :created_at, :updated_at, :deprecated_at],
methods: [:has_wiki, :material_id, :has_deerjikists] }.freeze
methods: [:material_id, :has_deerjikists] }.freeze
module_function
def base tag, language_code: 'ja'
tag.as_json(BASE).merge(name: name_for(tag, language_code),
tag.as_json(BASE).merge('has_wiki' => tag.has_description?(language_code:),
name: name_for(tag, language_code),
aliases: tag.snapshot_aliases(language_code),
parents: tag.parents.map { inline(_1, language_code:) })
end
def inline tag, language_code: 'ja'
tag.as_json(BASE).merge(name: name_for(tag, language_code), aliases: [], parents: [])
tag.as_json(BASE).merge('has_wiki' => tag.has_description?(language_code:),
name: name_for(tag, language_code), aliases: [], parents: [])
end
def many(tags, language_code: 'ja') = tags.map { |tag| base(tag, language_code:) }
def many tags, language_code: 'ja'
records = tags.to_a
ActiveRecord::Associations::Preloader.new(
records:, associations: [:tag_descriptions, { parents: :tag_descriptions }]).call
records.map { |tag| base(tag, language_code:) }
end
def name_for tag, language_code
tag.name(language_code) || (tag.system? ? tag.name('ja') : nil)
+29 -11
ファイルの表示
@@ -2,18 +2,16 @@
module WikiPageRepr
BASE = { methods: [:title, :deprecated_at] }.freeze
module_function
def base wiki_page
wiki_page.as_json(BASE)
end
def tag_description description
def tag_description locale, description
tag = description.tag
{ 'id' => description.id,
'tag_name_id' => tag.tag_name('ja')&.id,
'resource_type' => 'tag_description',
'tag_id' => tag.id,
'language_code' => description.language_code,
'tag_name_id' => tag.tag_name(locale.language_code)&.id,
'body' => description.body,
'created_user_id' => description.created_by_user_id,
'updated_user_id' => description.updated_by_user_id,
@@ -21,11 +19,31 @@ module WikiPageRepr
'updated_at' => description.updated_at,
'discarded_at' => nil,
'version_no' => description.version_no,
'title' => tag.name('ja'),
'title' => tag.name(locale.language_code),
'deprecated_at' => tag.deprecated_at }
end
def many wiki_pages
wiki_pages.map { |p| base(p) }
def tag_group tag_group
TagGroupRepr.tag_group(tag_group).merge('resource_type' => 'tag_group')
end
def autolink_tag locale, tag_id:, description_id:, name:, language:
{ resource_type: 'tag_description',
resource_id: description_id,
tag_id:,
tag_description_id: description_id,
name:,
language_code: language,
locale_code: locale.code_before_type_cast }
end
def autolink_group locale, id:, name:, language:
{ resource_type: 'tag_group',
resource_id: id,
tag_id: nil,
tag_description_id: nil,
name: "#{ TagGroupRepr::PREFIX }#{ name }",
language_code: language,
locale_code: locale.code_before_type_cast }
end
end
+105
ファイルの表示
@@ -0,0 +1,105 @@
require 'json'
require 'pathname'
class CertifiedTagNameTranslations
FORMAT = 'btrc-hub.certified-tag-name-translations.v1'.freeze
class Error < ArgumentError
;
end
def self.load! path, locale:
new(path, locale:).load!
end
def initialize path, locale:
@path = path
@locale = locale
end
def load!
document = read_document
unless document.is_a?(Hash) && document['format'] == FORMAT
fail!('invalid format or version')
end
source_code = document['source_locale']
target_code = document['target_locale']
unless target_code.is_a?(String) && target_code == Locale.codes.fetch(@locale.code)
fail!("target_locale #{ target_code.inspect } does not match the new Locale")
end
unless source_code.is_a?(String) && Locale.codes.value?(source_code)
fail!("invalid source_locale #{ source_code.inspect }")
end
source_locale = Locale.find_by(code: source_code)
fail!("source Locale #{ source_code.inspect } does not exist") unless source_locale
fail!('source_locale and target_locale must differ') if source_code == target_code
entries = validate_entries(document['entries'], source_code, target_code)
resolve_tags(entries, source_locale.language_code)
end
private
def read_document
unless (@path.is_a?(String) && !@path.include?("\0") &&
Pathname.new(@path).absolute?)
fail!('certified_tag_names_path must be an absolute file path')
end
fail!('JSON file does not exist or is not a regular file') unless File.file?(@path)
text = File.read(@path, encoding: Encoding::UTF_8)
fail!('JSON file is not valid UTF-8') unless text.valid_encoding?
JSON.parse(text)
rescue JSON::ParserError, SystemCallError, IOError, EncodingError => error
fail!("cannot read JSON: #{ error.message }")
end
def validate_entries entries, source_code, target_code
fail!('entries must be an array') unless entries.is_a?(Array)
sources = {}
targets = {}
entries.each_with_index.map do |entry, index|
fail!("entry #{ index + 1 } must be an object") unless entry.is_a?(Hash)
source = entry[source_code]
target = entry[target_code]
unless source.is_a?(String) && source.present? && target.is_a?(String) && target.present?
fail!("entry #{ index + 1 }: missing or empty #{ source_code }/#{ target_code } name")
end
fail!("duplicate source name #{ source.inspect }") if sources.key?(source)
fail!("duplicate target name #{ target.inspect }") if targets.key?(target)
sources[source] = true
targets[target] = true
[source, target]
end
end
def resolve_tags entries, source_language
source_names = entries.map(&:first)
# SQL may use a case/accent-insensitive collation; the Ruby keys are exact.
matches = TagName.where(language_code: source_language, primary_flg: true,
name: source_names).pluck(:name, :tag_id).group_by(&:first)
translations = {}
entries.each do |source, target|
rows = matches.fetch(source, [])
next if rows.empty?
unless rows.length == 1
fail!("source primary name #{ source.inspect } must match exactly one tag")
end
tag_id = rows.first.last
if translations.key?(tag_id)
fail!("multiple source primary names resolve to Tag##{ tag_id }: #{ source.inspect }")
end
translations[tag_id] = target
end
translations
end
def fail! message
raise Error, "Certified tag names #{@path.inspect}: #{ message }"
end
end
+98
ファイルの表示
@@ -0,0 +1,98 @@
class CountsAxes
GRAINS = ['year', 'quarter', 'month', 'week', 'day', 'hour'].freeze
def self.time_expression(expression, grain, timezone)
raise ArgumentError unless GRAINS.include?(grain)
connection = ApplicationRecord.connection
# Use the installed TZInfo database, including DST, rather than MySQL timezone tables.
first = Time.utc(1900)
last = Time.utc(2101)
offset = timezone.period_for_utc(first).utc_total_offset
clauses = timezone.transitions_up_to(last, first).filter_map do |transition|
next if transition.offset.utc_total_offset == offset
previous = offset
offset = transition.offset.utc_total_offset
"WHEN #{ expression } < #{ connection.quote(transition.at.to_time) } THEN #{ previous }"
end
seconds = clauses.empty? ? offset.to_s : "CASE #{ clauses.join(' ') } ELSE #{ offset } END"
local = "DATE_ADD(#{ expression }, INTERVAL (#{ seconds }) SECOND)"
case grain
when 'year' then "DATE_FORMAT(#{ local }, '%Y')"
when 'quarter' then "CONCAT(YEAR(#{ local }), '-Q', QUARTER(#{ local }))"
when 'month' then "DATE_FORMAT(#{ local }, '%Y-%m')"
when 'week' then "DATE_FORMAT(#{ local }, '%x-W%v')"
when 'day' then "DATE_FORMAT(#{ local }, '%Y-%m-%d')"
else "DATE_FORMAT(#{ local }, '%Y-%m-%d %H:00')"
end
end
attr_reader :rows, :columns, :all
def initialize(population)
@population = population
@definition = population.definition
@rows = parse(population.params.fetch('rows', '[]'))
@columns = parse(population.params.fetch('cols', '[]'))
@all = rows + columns
raise ArgumentError if all.length > 8 || all.map { _1['field'] }.uniq.length != all.length
end
def expressions
all.map do |axis|
field = axis.fetch('field')
case field
when 'tags'
"CASE WHEN tag_entry.tag_id IS NOT NULL " +
"THEN CONCAT('internal:', tag_entry.tag_id) " +
"ELSE CONCAT('external:', tag_entry.external_tag_id) END"
when 'changed_field' then 'changed.field_key'
else
expression = @definition.expression(field)
@definition.fields[field] == :datetime ? self.class.time_expression(
expression, axis.fetch('grain', 'month'), @population.timezone) : expression
end
end
end
def joins
result = ''
if all.any? { _1['field'] == 'tags' }
result += <<~SQL.squish
JOIN JSON_TABLE(CAST(v.tags_json AS JSON), '$[*]' COLUMNS (
tag_id BIGINT PATH '$.tag_id' NULL ON EMPTY,
external_tag_id BIGINT PATH '$.external_tag_id' NULL ON EMPTY)) tag_entry
SQL
end
if all.any? { _1['field'] == 'changed_field' }
fields = @definition.changes.map do |field|
"SELECT #{ ApplicationRecord.connection.quote(field) } AS field_key"
end.join(' UNION ALL ')
condition = @definition.changes.map do |field|
"(changed.field_key = '#{ field }' AND #{ @population.changed_sql(field) })"
end.join(' OR ')
result += " JOIN (#{ fields }) changed ON (#{ condition })"
end
result
end
private
def parse(value)
axes = JSON.parse(value)
raise ArgumentError unless axes.is_a?(Array)
axes.each do |axis|
raise ArgumentError unless axis.is_a?(Hash) && (axis.keys - ['field', 'grain']).empty?
raise ArgumentError unless @definition.axes(@population.history).include?(axis['field'])
if @definition.fields[axis['field']] == :datetime
raise ArgumentError unless GRAINS.include?(axis.fetch('grain', 'month'))
elsif axis.key?('grain')
raise ArgumentError
end
end
axes
end
end
+51
ファイルの表示
@@ -0,0 +1,51 @@
require 'tempfile'
require 'fileutils'
# Finish generation before reporting readiness; downloads never hold the full file in RAM.
class CountsExportDownload
EXPIRY = 10.minutes
def self.directory = Rails.root.join('tmp', 'counts_exports')
def self.verifier = Rails.application.message_verifier('counts_export_download')
def self.prepare(exporter, owner)
FileUtils.mkdir_p(directory, mode: 0o700)
Dir.glob(directory.join('counts-*.export')).each do |path|
File.unlink(path) if File.mtime(path) < Time.current - EXPIRY
rescue Errno::ENOENT
# Another request may have removed an expired download.
end
file = Tempfile.create(['counts-', '.part'], directory)
complete = false
path = file.path
begin
file.binmode
exporter.body.each { file.write(_1) }
file.close
finished_path = path.sub(/\.part\z/, '.export')
File.rename(path, finished_path)
path = finished_path
token = verifier.generate({ 'file' => File.basename(path),
'resource' => exporter.resource, 'format' => exporter.format,
'owner' => owner }, expires_in: EXPIRY, purpose: 'counts_export')
complete = true
token
ensure
file.close unless file.closed?
File.unlink(path) if !complete && File.exist?(path)
end
end
def self.find(token, owner)
metadata = verifier.verified(token, purpose: 'counts_export')
return unless metadata.is_a?(Hash) && metadata['owner'] == owner
return unless metadata['file']&.match?(/\Acounts-[a-zA-Z0-9_-]+\.export\z/)
return unless CountsResourceDefinition::DEFINITIONS.key?(metadata['resource'])
return unless ['json', 'csv'].include?(metadata['format'])
path = directory.join(metadata['file'])
return unless path.file?
metadata.merge('path' => path.to_s)
end
end
+229
ファイルの表示
@@ -0,0 +1,229 @@
require 'set'
# Export identifiers are stable; UI translations are supplied separately as column labels.
class CountsExportProjection
ROOT_KEYS = { 'event_type' => 'event_type', 'resource_id' => 'resource_id', 'version_no' => 'version_no',
'is_latest' => 'is_latest', 'thumbnail' => 'thumbnail_url',
'thumbnail_base' => 'thumbnail_base', 'title' => 'title', 'url' => 'url',
'tags' => 'tags', 'parents' => 'parents', 'video_ms' => 'video_ms',
'original_created_at' => 'original_created_at',
'original_created_from' => 'original_created_from',
'original_created_before' => 'original_created_before',
'user' => 'created_by_user', 'created_at' => 'created_at',
'name' => 'name', 'names' => 'names', 'category' => 'category',
'aliases' => 'aliases', 'deprecated' => 'deprecated' }.freeze
CHILD_KEYS = { 'id' => 'id', 'versionNo' => 'version_no', 'kind' => 'kind',
'name' => 'name', 'category' => 'category', 'sections' => 'sections',
'names' => 'names', 'aliases' => 'aliases',
'languageCode' => 'language', 'scriptCode' => 'script',
'autoGenerated' => 'auto_generated' }.freeze
NAME_FIELDS = ['languageCode', 'scriptCode', 'name', 'aliases', 'autoGenerated'].freeze
CHILDREN = {
'tags' => ['id', 'versionNo', 'kind', 'name', 'category', 'sections', 'names'],
'names' => NAME_FIELDS,
'aliases' => ['languageCode', 'name', 'scriptCode'],
'parents' => ['id'] }.freeze
attr_reader :columns, :headers
def initialize(params, resource:, summary:, tags:)
@params = params
@labels = JSON.parse(params.fetch('labels', '{}'))
raise ArgumentError unless @labels.is_a?(Hash) && @labels.length <= 100
raise ArgumentError unless @labels.all? { |key, value| valid_label?(key, value) }
@summary = summary
allowed = if summary
tags ? ['kind', 'tag_id', 'name', 'deprecated', 'count'] : ['group', 'count']
else
CountsQuery::COLUMNS.fetch(resource)
end
@columns = params.fetch('columns', allowed.join(',')).split(',')
raise ArgumentError if (@columns - allowed).any? || @columns.uniq != @columns
@headers = JSON.parse(params.fetch('headers', JSON.generate(@columns)))
unless (@headers.is_a?(Array) && @headers.length == @columns.length &&
@headers.all? { _1.is_a?(String) && _1.length <= 200 })
raise ArgumentError
end
@order = params.fetch('order', '').split(',')
@sorts = params.fetch('child_sort', '').split(',').map do |entry|
parts = entry.split(':', -1)
raise ArgumentError unless parts.length == 2
path, direction = parts
raise ArgumentError unless ['asc', 'desc'].include?(direction)
[path, direction]
end
@selected = params.key?('children') ? params['children'].split(',').to_set : nil
valid = CHILDREN.flat_map do |key, fields|
paths(fields, key)
end
raise ArgumentError if @selected && (@selected.to_a - valid).any?
allowed_sorts = valid.map do |path|
parts = path.split('.')
field = parts.pop
"#{ parts.join('.') }/#{ field }"
end
if ((@sorts.map(&:first) - allowed_sorts).any? ||
@sorts.map(&:first).uniq.length != @sorts.length)
raise ArgumentError
end
end
def metadata
@columns.each_with_index.map do |key, index|
{ key: @summary ? key : ROOT_KEYS.fetch(key), label: @headers[index] }
end
end
def project(row)
@columns.to_h do |key|
if @summary
value = key == 'group' ? { key: row[:key], name: row[:name] } : row[key.to_sym]
[key, value]
else
value = case key
when 'thumbnail' then row[:thumbnail_base]
when 'original_created_at'
{ from: row[:original_created_from], before: row[:original_created_before] }
else row[ROOT_KEYS.fetch(key).to_sym]
end
value = nested(value || [], CHILDREN.fetch(key), key) if CHILDREN.key?(key)
[ROOT_KEYS.fetch(key), value]
end
end
end
def csv_record(record, row)
record.map do |key, value|
key == 'group' ? row[:name] || row[:key] : csv_display(value, key)
end
end
private
def valid_label?(key, value)
key.is_a?(String) && value.is_a?(String) && key.length <= 100 && value.length <= 300
end
def csv_display(value, key)
return value ? @labels.fetch('latest', 'true') : '' if key == 'is_latest'
return @labels.fetch("kind:#{ value }", value) if key == 'kind'
return @labels.fetch("category:#{ value }", value) if key == 'category'
if ['language', 'script'].include?(key) && value.is_a?(Hash)
return @labels.fetch("#{ key }:#{ value[:code] }",
value[:name] || @labels.fetch('notRegistered', ''))
end
return @labels.fetch("boolean:#{ value }", value) if value == true || value == false
return value.map { csv_display(_1, key) } if value.is_a?(Array)
if value.is_a?(Hash)
return value.to_h do |field, item|
[@labels.fetch(field.to_s, field), csv_display(item, field.to_s)]
end
end
value
end
def fields_for(key)
return NAME_FIELDS if key == 'names'
return ['name', 'scriptCode'] if key == 'aliases'
[]
end
def paths(fields, prefix)
fields.flat_map do |key|
path = "#{ prefix }.#{ key }"
[path, *paths(fields_for(key), path)]
end
end
def fetch(row, key) = row.key?(key) ? row[key] : row[key.to_sym]
def named_code(row, kind)
code = fetch(row, "#{ kind }_code")
name = fetch(row, "#{ kind }_name")
{ code:, name: name.present? && name.to_s.downcase != code.to_s.downcase ? name : nil }
end
def stable(value)
case value
when Hash
value.stringify_keys.sort.to_h.transform_values { stable(_1) }
when Array
value.map { stable(_1) }
else
value
end
end
def sort_value(row, key)
if key == 'languageCode'
language = named_code(row, 'language')
return @labels.fetch("language:#{ language[:code] }", language[:name])
end
return named_code(row, 'script')[:name] if key == 'scriptCode'
fetch(row, CHILD_KEYS.fetch(key))
end
def comparison(left, right)
if left.is_a?(Array) && right.is_a?(Array) && left.length != right.length
return left.length <=> right.length
end
return left <=> right if left.is_a?(Numeric) && right.is_a?(Numeric)
if [true, false].include?(left) && [true, false].include?(right)
return (left ? 1 : 0) <=> (right ? 1 : 0)
end
a = left.is_a?(Hash) || left.is_a?(Array) ? JSON.generate(stable(left)) : left.to_s
b = right.is_a?(Hash) || right.is_a?(Array) ? JSON.generate(stable(right)) : right.to_s
a <=> b
end
def sorted(rows, prefix, fields)
sorts = @sorts.filter_map do |path, direction|
parent, key = path.split('/', 2)
next unless parent == prefix
raise ArgumentError unless fields.include?(key)
[key, direction]
end
rows.each_with_index.sort do |(left, a), (right, b)|
result = 0
sorts.each do |key, direction|
first = sort_value(left, key)
last = sort_value(right, key)
result = if first.nil? || last.nil?
first.nil? == last.nil? ? 0 : (first.nil? ? 1 : -1)
else
comparison(first, last) * (direction == 'asc' ? 1 : -1)
end
break unless result.zero?
end
result.zero? ? a <=> b : result
end.map(&:first)
end
def nested(rows, fields, prefix)
keys = fields.select { !@selected || @selected.include?("#{ prefix }.#{ _1 }") }
ordered = (@order + keys.map { "#{ prefix }.#{ _1 }" }).uniq
.filter_map { |path| keys.find { "#{ prefix }.#{ _1 }" == path } }
sorted(rows, prefix, fields).map do |row|
ordered.to_h do |key|
stable = CHILD_KEYS.fetch(key)
value = case key
when 'languageCode' then named_code(row, 'language')
when 'scriptCode' then named_code(row, 'script')
else fetch(row, stable)
end
if fields_for(key).any?
value = nested(value || [], fields_for(key), "#{ prefix }.#{ key }")
end
[stable, value]
end
end
end
end
+146
ファイルの表示
@@ -0,0 +1,146 @@
require 'csv'
# One read view and a fixed point for the entire download; only one page is held in memory.
class CountsExporter
EXTRA_KEYS = ['basis', 'format', 'resource', 'headers', 'children', 'order',
'summary_columns', 'nested_json', 'child_sort', 'labels'].freeze
attr_reader :format, :resource
def initialize(params, language_code:)
if params.key?('definition')
@table_export = CountsTableExport.new(params)
@format = @table_export.format
@resource = @table_export.resource
return
end
allowed = (CountsQuery::KEYS + CountsStateQuery::KEYS + EXTRA_KEYS).uniq
raise ArgumentError if (params.keys - allowed).any?
raise ArgumentError unless params.values.all? { _1.is_a?(String) }
@params = params
@language = language_code
@format = params.fetch('format', 'json')
@resource = params.fetch('resource', 'posts')
@mode = params.fetch('mode', 'summary')
@history = params.fetch('basis', 'state') == 'history'
raise ArgumentError unless ['json', 'csv'].include?(format)
raise ArgumentError unless ['posts', 'tags'].include?(resource)
raise ArgumentError unless ['summary', 'details'].include?(@mode)
raise ArgumentError unless ['state', 'history'].include?(params.fetch('basis', 'state'))
@query_params = params.except(*EXTRA_KEYS, 'columns', 'page', 'limit')
.merge('mode' => @mode, 'page' => '1', 'limit' => '100')
point = CountsStateQuery.new(@query_params.slice('at').merge('resource' => resource),
language_code: @language).at
@query_params['at'] = point.iso8601(6)
@query_params['axis'] ||= !@history && resource == 'posts' ? 'tags' : 'month'
@projection = CountsExportProjection.new(params, resource:,
summary: @mode == 'summary',
tags: !@history && @query_params['axis'] == 'tags')
if @history
@query_params.delete('kind')
history_query
else
@query_params.delete('latest')
@state_query = CountsStateQuery.new(@query_params.merge('resource' => resource),
language_code: @language).validate_query!
end
end
def body
return @table_export.body if @table_export
Enumerator.new do |output|
ApplicationRecord.transaction(isolation: :repeatable_read) do
first_row = true
started = false
pages do |page|
unless started
meta = { resource:, mode: @mode, basis: @history ? 'history' : 'state',
at: @query_params['at'], boundary: 'inclusive',
filters: @query_params.except('sort', 'page', 'limit', 'mode'),
sort: export_sort,
child_sort: sort_metadata(@params.fetch('child_sort', '')),
columns: @projection.metadata,
children: @params['children']&.split(','),
field_order: @params.fetch('order', '').split(','), timezone: 'UTC',
count: @mode == 'summary' ? page[:group_count] : page[:count],
totals: page[:totals], coverage: page[:coverage] }
output << (format == 'json' ? "{\"metadata\":#{ meta.to_json },\"data\":[" : "\uFEFF")
if format == 'csv'
output << CSV.generate_line(
@projection.headers.map { csv_value(_1) }, row_sep: "\r\n")
end
started = true
end
(page[:versions] || page[:state_rows] || page[:groups] || []).each do |row|
record = @projection.project(row)
if format == 'json'
output << ',' unless first_row
output << record.to_json
else
values = @projection.csv_record(record, row).map { csv_value(_1) }
output << CSV.generate_line(values, row_sep: "\r\n")
end
first_row = false
end
end
output << ']}' if format == 'json'
end
end
end
private
def export_sort
sort_metadata(@query_params.fetch('sort', default_sort))
end
def sort_metadata(value)
value.split(',').map do |entry|
key, direction = entry.split(':', 2)
{ key:, direction: }
end
end
def default_sort
return 'created_at:desc' if @mode == 'details'
!@history && @query_params['axis'] == 'tags' ? 'count:desc' : 'group:asc'
end
def history_query
scope = resource == 'posts' ? PostVersion.with_previous_version : TagVersion.all
CountsQuery.new(scope, resource, @query_params)
end
def pages
unless @history
@state_query.each_page { yield _1 }
return
end
query = history_query
total = @mode == 'summary' ? nil : query.count
loop do
result = if @mode == 'summary'
query.summary(include_count: @query_params['page'] == '1')
else
{ count: total,
versions: CountsVersionRepr.many(query.details, resource:,
language_code: @language,
at: Time.iso8601(@query_params['at'])) }
end
total ||= result[:group_count]
yield result
break if @query_params['page'].to_i * 100 >= total
@query_params['page'] = (@query_params['page'].to_i + 1).to_s
query = history_query
end
end
def csv_value(value)
text = value.is_a?(Hash) || value.is_a?(Array) ? JSON.generate(value) : value.to_s
text.match?(/\A\s*[=+@-]/) ? "'#{ text }" : text
end
end
+208
ファイルの表示
@@ -0,0 +1,208 @@
# Population first, then filters. Previous versions are joined before period filtering.
class CountsPopulation
attr_reader :definition, :params, :at, :history, :scope, :timezone
def initialize(params)
@params = params
@definition = CountsResourceDefinition.new(params.fetch('resource', 'posts'))
@history = params.fetch('definition', 'latest') == 'history'
raise ArgumentError unless ['latest', 'history'].include?(params.fetch('definition', 'latest'))
@timezone = TZInfo::Timezone.get(params.fetch('timezone', 'UTC'))
@at = timestamp(params['execution_at']) || Time.current.utc
raise ArgumentError if at > Time.current + 1.second
@scope = definition.model.all
make_population
apply_filters
rescue TZInfo::InvalidTimezoneIdentifier
raise ArgumentError
end
def timestamp(value, end_date: false)
return nil if value.nil? || value == ''
raise ArgumentError unless value.is_a?(String)
if value.match?(/\A\d{4}-\d{2}-\d{2}\z/)
date = Date.iso8601(value)
date += 1 if end_date
timezone.local_to_utc(Time.utc(date.year, date.month, date.day))
else
raise ArgumentError unless value.match?(/(?:Z|[+-]\d{2}:\d{2})\z/)
time = Time.iso8601(value).utc
raise ArgumentError unless time.year.between?(1900, 2100)
time
end
rescue TZInfo::AmbiguousTime, TZInfo::PeriodNotFound, Date::Error
raise ArgumentError
end
def latest_sql(row = 'v', cutoff = nil)
cutoff ||= history ? at : timestamp(params['at']) || at
keys = definition.keys.map { "newer.#{ _1 } = #{ row }.#{ _1 }" }.join(' AND ')
bound = definition.model.connection.quote(cutoff)
<<~SQL.squish
NOT EXISTS (SELECT 1 FROM #{ definition.table } newer
WHERE #{ keys } AND newer.version_no > #{ row }.version_no
AND newer.#{ definition.time_column } <= #{ bound })
SQL
end
def changed_sql(key)
column = definition.column(key)
"(previous.id IS NOT NULL AND NOT (v.#{ column } <=> previous.#{ column }))"
end
private
def make_population
table = definition.table
@scope = @scope.from("#{ table } v").select('v.*')
keys = definition.keys.map { "previous.#{ _1 } = v.#{ _1 }" }.join(' AND ')
@scope = @scope.joins("LEFT JOIN #{ table } previous ON #{ keys } " +
'AND previous.version_no = v.version_no - 1')
@scope = @scope.where("v.#{ definition.time_column } <= ?", at)
unless history
point = timestamp(params['at']) || at
raise ArgumentError if point > at
@scope = @scope.where("v.#{ definition.time_column } <= ?", point)
.where(Arel.sql(latest_sql('v', point))).where.not('v.event_type' => 'discard')
return
end
first = timestamp(params['from'])
last = timestamp(params['through'], end_date: true)
raise ArgumentError if first && last && first > last
@scope = @scope.where('v.created_at >= ?', first) if first
if last
operator = params['through'].length == 10 ? '<' : '<='
@scope = @scope.where("v.created_at #{ operator } ?", last)
end
end
def apply_filters
['resource_id', 'user'].each do |key|
value = params[key == 'user' ? 'user_id' : key]
next if value.nil? || value == ''
raise ArgumentError unless value.match?(/\A[0-9]+\z/)
@scope = @scope.where("#{ definition.expression(key) } = ?", value.to_i)
end
apply_tags if params['tags'].present? || params['tag'].present?
conditions = JSON.parse(params.fetch('conditions', '[]'))
raise ArgumentError unless conditions.is_a?(Array) && conditions.length <= 100
conditions.each { apply_condition(_1) }
drills = JSON.parse(params.fetch('drill', '[]'))
raise ArgumentError unless drills.is_a?(Array) && drills.length <= 100
drills.each do |entry|
key = entry.fetch('field')
if key == 'tags'
@scope = @scope.where(snapshot_membership(entry.fetch('value')))
elsif key == 'changed_field'
raise ArgumentError unless history && definition.changes.include?(entry['value'])
@scope = @scope.where(Arel.sql(changed_sql(entry['value'])))
elsif definition.fields[key] == :datetime && entry['grain']
@scope = @scope.where(definition.expression(key) => nil) if entry['value'].nil?
if entry['value']
expression = CountsAxes.time_expression(definition.expression(key),
entry['grain'], timezone)
@scope = @scope.where("#{ expression } = ?", entry['value'])
end
else
raise ArgumentError unless definition.axes(history).include?(key)
@scope = @scope.where(definition.expression(key) => entry['value'])
end
end
end
def apply_condition(entry)
raise ArgumentError unless entry.is_a?(Hash)
raise ArgumentError if (entry.keys - ['field', 'operator', 'value', 'end', 'blank']).any?
key = entry.fetch('field')
type = definition.fields.fetch(key) { raise ArgumentError }
operator = entry.fetch('operator')
raise ArgumentError unless CountsResourceDefinition::OPERATORS.fetch(type, []).include?(operator)
raise ArgumentError unless [true, false, nil].include?(entry['blank'])
expression = definition.expression(key)
blank = type == :string || type == :enum ?
"(#{ expression } IS NULL OR #{ expression } = '')" : "#{ expression } IS NULL"
if entry['blank']
@scope = @scope.where(Arel.sql(blank))
return
end
values = [entry['value'], entry['end']]
raise ArgumentError unless values.all? { _1.nil? || _1.is_a?(String) }
return if values.all? { _1.nil? || _1 == '' }
parsed = values.map do |value|
next nil if value.nil? || value == ''
case type
when :integer
raise ArgumentError unless value.match?(/\A-?\d+\z/)
value.to_i
when :datetime then timestamp(value)
else value
end
end
if operator == 'range'
@scope = @scope.where("#{ expression } >= ?", parsed[0]) if parsed[0]
@scope = @scope.where("#{ expression } <= ?", parsed[1]) if parsed[1]
return
end
return unless parsed[0]
sql_operator = { 'eq' => '=', 'ne' => '<>', 'gt' => '>', 'lt' => '<',
'ge' => '>=', 'le' => '<=', 'like' => 'LIKE' }.fetch(operator)
sql = "#{ expression } #{ sql_operator } ?"
sql = "(#{ sql } OR #{ blank })" if operator == 'ne'
@scope = @scope.where(sql, parsed[0])
end
def snapshot_membership(value)
kind, identifier = value.to_s.split(':', 2)
raise ArgumentError unless ['internal', 'external'].include?(kind)
raise ArgumentError unless identifier&.match?(/\A[1-9]\d*\z/)
key = kind == 'internal' ? 'tag_id' : 'external_tag_id'
definition.model.sanitize_sql_array([
"JSON_CONTAINS(v.tags_json, JSON_OBJECT('#{ key }', CAST(? AS UNSIGNED)))",
identifier.to_i])
end
def apply_tags
raise ArgumentError unless definition.resource == 'posts'
if params['tag'].present?
@scope = @scope.where(snapshot_membership(
"#{ params.fetch('kind', 'internal') }:#{ params['tag'] }"))
end
return unless params['tags'].present?
locale = Locale.find_by!(code: params.fetch('locale', 'ja'))
refs = PostTagSearch.parse_references(params['tag_refs'], params['tags'].split)
raw = params['tags']
literals = PostTagSearch.literals(locale, raw, references: refs)
predicates = literals.map do |literal|
internal, external = PostTagSearch.references(locale, literal)
parts = Array(internal).map { snapshot_membership("internal:#{ _1 }") } +
Array(external).map { snapshot_membership("external:#{ _1 }") }
expression = parts.empty? ? 'FALSE' : "(#{ parts.join(' OR ') })"
literal[:negative] ? "NOT #{ expression }" : expression
end
return if predicates.empty?
any = params['match'] == 'any' || raw.split.any? { _1.downcase == 'type:or' }
@scope = @scope.where(Arel.sql("(#{ predicates.join(any ? ' OR ' : ' AND ') })"))
end
end
+208
ファイルの表示
@@ -0,0 +1,208 @@
# Read-only history queries. SQL identifiers and expressions are server-owned.
class CountsQuery
AXES = ['month', 'user'].freeze
COMMON_COLUMNS = ['resource_id', 'version_no', 'is_latest', 'event_type', 'user', 'created_at'].freeze
COLUMNS = {
'posts' => COMMON_COLUMNS + [
'thumbnail', 'thumbnail_base', 'title', 'url', 'tags', 'parents', 'video_ms',
'original_created_at', 'original_created_from', 'original_created_before'],
'tags' => COMMON_COLUMNS + [
'name', 'names', 'category', 'aliases', 'parents', 'deprecated'] }.freeze
KEYS = [
'locale', 'mode', 'axis', 'page', 'limit', 'from', 'before', 'latest',
'user_id', 'resource_id', 'tag', 'columns', 'sort', 'at'].freeze
attr_reader :page, :limit, :mode, :axis
def initialize(scope, resource, params, latest_expression: nil)
@table = scope.klass.table_name
@resource = resource
@id_column = resource == 'posts' ? 'post_id' : 'tag_id'
@params = params
@latest_expression = latest_expression
@scope = scope.except(:select, :order).joins(
"LEFT JOIN users counts_actor ON counts_actor.id = #{ @table }.created_by_user_id")
raise ArgumentError unless (params.keys - KEYS).empty?
raise ArgumentError unless params.values.all? { _1.is_a?(String) }
@mode = params.fetch('mode', 'summary')
@axis = params.fetch('axis', 'month')
raise ArgumentError unless ['summary', 'details'].include?(mode) && AXES.include?(axis)
@page = integer('page', 1)
@limit = integer('limit', 20)
raise ArgumentError if limit > 100 || page > 1_000_000
columns = params.fetch('columns', COLUMNS.fetch(resource).join(',')).split(',')
if columns.empty? || columns.uniq != columns || (columns - COLUMNS.fetch(resource)).any?
raise ArgumentError
end
@sort = parse_sort
apply_filters
end
def relation = @scope
def count = @scope.count
def latest_expression
return @latest_expression if @latest_expression
at = timestamp('at')
cutoff = at ? "AND newer.created_at <= #{ @scope.klass.connection.quote(at) }" : ''
<<~SQL.squish
NOT EXISTS (SELECT 1 FROM #{ @table } newer
WHERE newer.#{ @id_column } = #{ @table }.#{ @id_column }
AND newer.version_no > #{ @table }.version_no #{ cutoff })
SQL
end
def details
@scope.select("#{ @table }.*", Arel.sql("(#{ latest_expression }) AS counts_latest"))
.order(Arel.sql(order_sql(detail_expressions, ["#{ @table }.id DESC"])))
.limit(limit).offset((page - 1) * limit)
end
def summary(include_count: true)
expression = axis == 'month' ? "DATE_FORMAT(#{ @table }.created_at, '%Y-%m')" :
"#{ @table }.created_by_user_id"
group_order = axis == 'user' ? 'MAX(counts_actor.name)' : expression
sql = @scope.group(Arel.sql(expression)).select(Arel.sql(
"#{ expression } AS group_key, COUNT(*) AS record_count, " +
"#{ group_order } AS group_name")).to_sql
connection = @scope.klass.connection
source = "FROM (#{ sql }) counts_groups"
groups = include_count ? connection.select_value("SELECT COUNT(*) #{ source }") : nil
expressions = { 'group' => ['group_name'], 'count' => ['record_count'] }
order = order_sql(expressions, ['group_key IS NULL ASC', 'group_key ASC'])
rows = connection.select_all(
"SELECT group_key, group_name, record_count #{ source } " +
"ORDER BY #{ order } LIMIT #{ limit } OFFSET #{ (page - 1) * limit }")
{ group_count: groups&.to_i,
groups: rows.map do |row|
{ key: row['group_key'],
name: axis == 'user' ? row['group_name'] : nil,
count: row['record_count'].to_i }
end }
end
private
def parse_sort
value = @params.fetch('sort', mode == 'details' ? 'created_at:desc' : 'group:asc')
return [] if value.empty?
values = value.split(',', -1).map { _1.split(':', -1) }
allowed = mode == 'details' ? COLUMNS.fetch(@resource) - ['thumbnail'] : ['group', 'count']
invalid = values.any? do |key, direction, *extra|
extra.any? || !allowed.include?(key) || !['asc', 'desc'].include?(direction)
end
if (values.empty? || values.length > allowed.length || invalid ||
values.map(&:first).uniq.length != values.length)
raise ArgumentError
end
values
end
# NULL always sorts last, including descending keys. The final key is unique.
def order_sql(expressions, final)
(@sort.flat_map do |key, direction|
expressions.fetch(key).flat_map do |expression|
["(#{ expression }) IS NULL ASC", "#{ expression } #{ direction.upcase }"]
end
end + final).join(', ')
end
def detail_expressions
common = { 'resource_id' => ["#{ @table }.#{ @id_column }"],
'version_no' => ["#{ @table }.version_no"],
'is_latest' => ["(#{ latest_expression })"],
'user' => ['counts_actor.name', "#{ @table }.created_by_user_id"],
'created_at' => ["#{ @table }.created_at"] }
if @resource == 'posts'
fields = [
'title', 'url', 'thumbnail_base', 'video_ms',
'original_created_from', 'original_created_before']
common.merge(fields.to_h { [_1, ["#{ @table }.#{ _1 }"]] }).merge(
'original_created_at' => [
'post_versions.original_created_from', 'post_versions.original_created_before'],
'tags' => json_order('post_versions.tags_json'),
'parents' => list_order('post_versions.parent_post_ids'))
else
name = "JSON_UNQUOTE(JSON_EXTRACT(tag_versions.names, #{ locale_name_path }))"
common.merge(
'name' => ["COALESCE(#{ name }, tag_versions.name)"],
'names' => json_order('tag_versions.names'),
'category' => ['tag_versions.category'],
'aliases' => json_order(
"JSON_EXTRACT(tag_versions.names, '$.*.aliases[*]')"),
'parents' => list_order('tag_versions.parent_tag_ids'),
'deprecated' => ['tag_versions.deprecated_at'])
end
end
def locale_name_path
language = Locale.find_by(code: @params.fetch('locale', 'ja'))&.language_code || 'ja'
path = "$.#{ JSON.generate(language) }.name"
@scope.klass.connection.quote(path)
end
# Multi-valued parent cells sort by cardinality, then binary snapshot text.
def json_order(column)
[
"COALESCE(JSON_LENGTH(#{ column }), 0)",
"COALESCE(CAST(#{ column } AS CHAR), '[]') COLLATE utf8mb4_bin"]
end
def list_order(column)
[
"CASE WHEN #{ column } = '' THEN 0 ELSE " +
"1 + LENGTH(#{ column }) - LENGTH(REPLACE(#{ column }, ' ', '')) END",
"#{ column } COLLATE utf8mb4_bin"]
end
def integer(key, default = nil)
value = @params[key].presence
return default if value.nil? && !default.nil?
raise ArgumentError unless value&.match?(/\A[1-9][0-9]{0,9}\z/)
value.to_i
end
def timestamp(key)
value = @params[key].presence
return nil unless value
pattern = /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(?::\d{2}(?:\.\d{1,6})?)?Z\z/
raise ArgumentError unless value.match?(pattern)
Date.iso8601(value.first(10))
Time.iso8601(value.length == 17 ? value.sub(/Z\z/, ':00Z') : value)
end
def apply_filters
at = timestamp('at')
first = timestamp('from')
last = timestamp('before')
raise ArgumentError if first && last && first >= last
raise ArgumentError unless ['0', '1'].include?(@params.fetch('latest', '1'))
@scope = @scope.where("#{ @table }.created_at <= ?", at) if at
@scope = @scope.where(Arel.sql(latest_expression)) unless @params['latest'] == '0'
@scope = @scope.where("#{ @table }.created_at >= ?", first) if first
@scope = @scope.where("#{ @table }.created_at < ?", last) if last
if @params['resource_id'].present?
@scope = @scope.where(@table => { @id_column => integer('resource_id') })
end
if @params['user_id'].present?
user = @params['user_id'] == 'none' ? nil : integer('user_id')
@scope = @scope.where(@table => { created_by_user_id: user })
end
return unless @params['tag'].present?
raise ArgumentError unless @resource == 'posts'
id = integer('tag')
@scope = @scope.where(
"JSON_CONTAINS(post_versions.tags_json, JSON_OBJECT('tag_id', CAST(? AS UNSIGNED))) OR " +
"JSON_CONTAINS(prev.tags_json, JSON_OBJECT('tag_id', CAST(? AS UNSIGNED)))", id, id)
end
end
+65
ファイルの表示
@@ -0,0 +1,65 @@
# Server-owned version identities, attributes and field types. No client SQL is accepted.
class CountsResourceDefinition
COMMON = { 'resource_id' => :integer, 'version_no' => :integer,
'event_type' => :enum, 'user' => :integer, 'created_at' => :datetime }.freeze
DEFINITIONS = {
'posts' => [PostVersion, ['post_id'], {
'title' => :string, 'url' => :string, 'thumbnail_base' => :string,
'tags_json' => :json, 'parent_post_ids' => :string, 'video_ms' => :integer,
'original_created_from' => :datetime, 'original_created_before' => :datetime }],
'tags' => [TagVersion, ['tag_id'], {
'name' => :string, 'names' => :json, 'category' => :enum, 'aliases' => :string,
'parent_tag_ids' => :string, 'deprecated_at' => :datetime }],
'nico_tags' => [NicoTagVersion, ['tag_id'], {
'name' => :string, 'linked_tags' => :string }],
'tag_descriptions' => [TagDescriptionVersion, ['tag_id', 'language_code'], {
'language_code' => :enum, 'body' => :string,
'source_language_code' => :enum, 'source_version_no' => :integer }],
'tag_groups' => [TagGroupVersion, ['tag_group_id'], {
'name' => :string, 'description' => :string,
'members' => :json, 'placements' => :json }],
'materials' => [MaterialVersion, ['material_id'], {
'url' => :string, 'parent_id' => :integer, 'tag_id' => :integer,
'tag_name' => :string, 'tag_category' => :enum,
'discarded_at' => :datetime, 'updated_at' => :datetime,
'updated_by_user_id' => :integer, 'export_paths_json' => :json,
'file_blob_id' => :integer, 'file_filename' => :string,
'file_content_type' => :string, 'file_byte_size' => :integer,
'file_checksum' => :string, 'file_sha256' => :string,
'source_kind' => :enum, 'source_uri' => :string, 'source_path' => :string,
'source_file_id' => :string, 'normalized_source_key' => :string }] }.freeze
OPERATORS = {
string: ['eq', 'ne', 'like'], enum: ['eq', 'ne'], boolean: ['eq', 'ne'],
integer: ['eq', 'ne', 'gt', 'lt', 'ge', 'le', 'range'],
datetime: ['eq', 'ne', 'gt', 'lt', 'ge', 'le', 'range'] }.freeze
attr_reader :model, :keys, :fields, :resource
def initialize(resource)
@resource = resource
@model, @keys, attributes = DEFINITIONS.fetch(resource) { raise ArgumentError }
@fields = COMMON.merge(attributes)
end
def table = model.table_name
def time_column = 'created_at'
def column(key)
raise ArgumentError unless fields.key?(key)
{ 'resource_id' => keys.first, 'user' => 'created_by_user_id' }.fetch(key, key)
end
def expression(key, row = 'v') = "#{ row }.#{ column(key) }"
def changes = fields.keys - COMMON.keys - ['updated_at', 'updated_by_user_id']
def axes(history)
fields.reject { |_key, type| type == :json }.keys +
(resource == 'posts' ? ['tags'] : []) + (history ? ['changed_field'] : [])
end
def metadata
fields.map do |key, type|
{ key:, type:, nullable: model.columns_hash.fetch(column(key)).null,
operators: OPERATORS.fetch(type, []) }
end
end
end
+366
ファイルの表示
@@ -0,0 +1,366 @@
# Restores one recorded version per resource before applying record filters.
class CountsStateQuery
KEYS = ['locale', 'resource', 'mode', 'at', 'from', 'before', 'grain',
'kind', 'tag', 'page', 'limit', 'sort', 'axis', 'user_id', 'resource_id'].freeze
MAX_PERIODS = 366
attr_reader :at
def initialize(params, language_code:)
raise ArgumentError unless (params.keys - KEYS).empty?
raise ArgumentError unless params.values.all? { _1.is_a?(String) }
@params = params
@connection = ApplicationRecord.connection
@language = language_code
@resource = params.fetch('resource', 'posts')
@mode = params.fetch('mode', 'summary')
@inclusive = ['summary', 'details'].include?(@mode)
@axis = params.fetch('axis', @resource == 'posts' ? 'tags' : 'month')
@kind = params.fetch('kind', 'internal')
@grain = params.fetch('grain', 'month')
raise ArgumentError unless ['posts', 'tags'].include?(@resource)
raise ArgumentError unless ['summary', 'details', 'state', 'trend', 'snapshot'].include?(@mode)
raise ArgumentError unless ['internal', 'external', 'all'].include?(@kind)
raise ArgumentError unless ['day', 'month', 'year'].include?(@grain)
raise ArgumentError unless ['tags', 'month', 'user'].include?(@axis)
if @mode == 'summary' && @axis == 'tags' && @resource != 'posts'
raise ArgumentError
end
@page = integer('page', 1)
@limit = integer('limit', 20)
@tag = integer('tag') if params['tag'].present?
raise ArgumentError if @limit > 100 || @page > 1_000_000
raise ArgumentError if @tag && @kind == 'all'
raise ArgumentError if @mode == 'trend' && !@tag
@now = Time.current.utc
@at = timestamp('at') || @now
raise ArgumentError if @at > @now
end
def call(metadata: true)
@metadata = metadata
# Keep totals, rows, and detail pages on one repeatable-read database snapshot.
ApplicationRecord.transaction do
unless @history_loaded
@post_start = PostVersion.minimum(:created_at)
@tag_start = TagVersion.minimum(:created_at)
@history_loaded = true
end
result = { page: @page, limit: @limit, at: @at.iso8601(6),
coverage: { basis: 'recorded_snapshots', exact: false,
post_history_from: @post_start&.iso8601(6),
tag_history_from: @tag_start&.iso8601(6) } }
case @mode
when 'trend'
result.merge(trend)
when 'summary'
result.merge(metadata ? { totals: totals } : {}).merge(summary)
when 'details', 'snapshot'
result.merge(metadata ? { totals: totals } : {}).merge(details)
else
result.merge(metadata ? { totals: totals } : {}).merge(listing)
end
end
end
def each_page
return enum_for(:each_page) unless block_given?
@page = 1
first = call
yield first
total = ['summary', 'state', 'trend'].include?(@mode) ? first[:group_count] : first[:count]
while @page * @limit < total
@page += 1
yield call(metadata: false)
end
end
def validate_query!
filtered_scope(@resource)
case @mode
when 'summary'
if @axis == 'tags'
order_sql({ 'name' => 'name', 'count' => 'post_count', 'tag_id' => 'tag_id',
'kind' => 'kind', 'deprecated' => 'deprecated_at' }, 'count:desc', [])
else
record_query('summary')
end
when 'details', 'snapshot'
record_query('details')
end
self
end
private
def quote(value) = @connection.quote(value)
def integer(key, default = nil)
value = @params[key]
return default if value.blank? && default
raise ArgumentError unless value&.match?(/\A[1-9][0-9]{0,9}\z/)
value.to_i
end
def timestamp(key)
value = @params[key].presence
return nil unless value
pattern = /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(?::\d{2}(?:\.\d{1,6})?)?Z\z/
raise ArgumentError unless value.match?(pattern)
Date.iso8601(value.first(10))
Time.iso8601(value.length == 17 ? value.sub(/Z\z/, ':00Z') : value).utc
end
# Version numbers order equal-timestamp writes. Do not filter tags before this selection.
def latest_condition(table, row, boundary)
key = table == 'post_versions' ? 'post_id' : 'tag_id'
<<~SQL.squish
#{ row }.created_at #{ @inclusive ? '<=' : '<' } #{ boundary } AND NOT EXISTS (
SELECT 1 FROM #{ table } newer
WHERE newer.#{ key } = #{ row }.#{ key }
AND newer.created_at #{ @inclusive ? '<=' : '<' } #{ boundary }
AND newer.version_no > #{ row }.version_no)
SQL
end
def state_sql(table, boundary, live: false)
sql = "SELECT v.* FROM #{ table } v WHERE #{ latest_condition(table, 'v', boundary) }"
live ? "#{ sql } AND v.event_type <> 'discard'" : sql
end
def membership(row)
key = @kind == 'external' ? 'external_tag_id' : 'tag_id'
"JSON_CONTAINS(#{ row }.tags_json, JSON_OBJECT('#{ key }', #{ @tag }))"
end
# Deprecation keeps a tag in existence. Only an explicit discard removes it.
def tag_visible(boundary)
return '1 = 1' if @kind == 'external'
<<~SQL.squish
NOT EXISTS (SELECT 1 FROM tag_versions t
WHERE t.tag_id = #{ @tag } AND t.event_type = 'discard'
AND #{ latest_condition('tag_versions', 't', boundary) })
SQL
end
def ctes
boundary = quote(@at)
<<~SQL
WITH post_at AS (#{ filtered_scope('posts').to_sql }),
tag_at AS (#{ state_sql('tag_versions', boundary) }),
nico_at AS (#{ state_sql('nico_tag_versions', boundary) }),
pairs AS (
SELECT DISTINCT p.post_id,
CASE WHEN entry.tag_id IS NULL THEN 'external' ELSE 'internal' END AS kind,
COALESCE(entry.tag_id, entry.external_tag_id) AS tag_id
FROM post_at p
JOIN JSON_TABLE(CAST(p.tags_json AS JSON), '$[*]' COLUMNS (
tag_id BIGINT PATH '$.tag_id' NULL ON EMPTY ERROR ON ERROR,
external_tag_id BIGINT PATH '$.external_tag_id' NULL ON EMPTY ERROR ON ERROR
)) entry
LEFT JOIN tag_at t ON t.tag_id = entry.tag_id
WHERE COALESCE(entry.tag_id, entry.external_tag_id) IS NOT NULL
AND (t.event_type IS NULL OR t.event_type <> 'discard')
)
SQL
end
def totals
row = @connection.select_one(<<~SQL)
#{ ctes }
SELECT (SELECT COUNT(*) FROM post_at) AS posts,
(SELECT COUNT(*) FROM (#{ filtered_scope('tags').to_sql }) counted_tags) AS tags,
(SELECT COUNT(*) FROM tag_at WHERE event_type <> 'discard'
AND deprecated_at IS NOT NULL) AS deprecated_tags,
(SELECT COUNT(*) FROM pairs) AS assignments,
(SELECT COUNT(*) FROM pairs WHERE kind = 'external') AS external_assignments
SQL
row.transform_values!(&:to_i)
row.merge('posts' => available?(@post_start, @at) ? row['posts'] : nil,
'assignments' => available?(@post_start, @at) ? row['assignments'] : nil,
'external_assignments' => available?(@post_start, @at) ?
row['external_assignments'] : nil,
'tags' => available?(@tag_start, @at) ? row['tags'] : nil,
'deprecated_tags' => available?(@tag_start, @at) ? row['deprecated_tags'] : nil)
end
def available?(first, boundary)
first && (@inclusive ? first <= boundary : first < boundary)
end
def listing
path = quote("$.#{ JSON.generate(@language) }.name")
catalog = <<~SQL
, tag_counts AS (
SELECT membership.kind, membership.tag_id, COUNT(*) AS post_count
FROM pairs membership GROUP BY membership.kind, membership.tag_id
), catalog AS (
SELECT 'internal' AS kind, tag_id,
COALESCE(JSON_UNQUOTE(JSON_EXTRACT(names, #{ path })), name) AS name,
deprecated_at FROM tag_at WHERE event_type <> 'discard'
UNION ALL
SELECT tc.kind, tc.tag_id, external_version.name, NULL AS deprecated_at
FROM tag_counts tc
LEFT JOIN tag_at internal_version
ON tc.kind = 'internal' AND internal_version.tag_id = tc.tag_id
LEFT JOIN nico_at external_version
ON tc.kind = 'external' AND external_version.tag_id = tc.tag_id
WHERE tc.kind = 'external' OR internal_version.tag_id IS NULL
)
SQL
filters = []
filters << "c.kind = #{ quote(@kind) }" unless @kind == 'all'
filters << "c.tag_id = #{ @tag }" if @tag
where = filters.any? ? "WHERE #{ filters.join(' AND ') }" : ''
source = 'FROM catalog c LEFT JOIN tag_counts n ON n.kind = c.kind AND n.tag_id = c.tag_id'
prefix = "#{ ctes } #{ catalog }"
total = if @metadata
@connection.select_value("#{ prefix } SELECT COUNT(*) #{ source } #{ where }").to_i
end
order = order_sql({ 'name' => 'c.name', 'count' => 'COALESCE(n.post_count, 0)',
'tag_id' => 'c.tag_id', 'kind' => 'c.kind',
'deprecated' => 'c.deprecated_at' }, 'count:desc',
['c.kind ASC', 'c.tag_id ASC'])
rows = @connection.select_all(<<~SQL)
#{ prefix }
SELECT c.kind, c.tag_id, c.name, c.deprecated_at,
COALESCE(n.post_count, 0) AS post_count #{ source } #{ where }
ORDER BY #{ order } LIMIT #{ @limit } OFFSET #{ (@page - 1) * @limit }
SQL
{ count: total, group_count: total, state_rows: rows.map do |row|
{ kind: row['kind'], tag_id: row['tag_id'], name: row['name'],
deprecated: row['deprecated_at']&.iso8601,
count: available?(@post_start, @at) ? row['post_count'].to_i : nil,
at: @at.iso8601(6) }
end }
end
# All record filters apply after selecting the latest recorded version at the point.
def restored_scope(resource)
model = resource == 'posts' ? PostVersion : TagVersion
table = model.table_name
scope = model.where(Arel.sql(latest_condition(table, table, quote(@at))))
.where.not(event_type: 'discard')
if @tag
scope = if resource == 'posts'
scope.where("#{ membership(table) } AND #{ tag_visible(quote(@at)) }")
elsif @kind == 'internal'
scope.where(tag_id: @tag)
else
scope.none
end
end
scope
end
def filtered_scope(resource)
params = @params.slice('from', 'before', 'user_id', 'resource_id')
.merge('mode' => 'details', 'latest' => '0', 'sort' => '')
CountsQuery.new(restored_scope(resource), resource, params,
latest_expression: '1').relation
end
def record_query(mode)
params = @params.slice('locale', 'sort', 'page', 'limit')
.merge('mode' => mode, 'latest' => '0', 'axis' => @axis == 'tags' ? 'month' : @axis,
'page' => @page.to_s, 'limit' => @limit.to_s)
CountsQuery.new(filtered_scope(@resource), @resource, params, latest_expression: '1')
end
def summary
return listing if @axis == 'tags'
query = record_query('summary')
{ count: @metadata ? query.count : nil }.merge(query.summary(include_count: @metadata))
end
def details
query = record_query('details')
{ count: @metadata ? query.count : nil,
versions: CountsVersionRepr.many(query.details, resource: @resource,
language_code: @language, at: @at,
inclusive: @inclusive) }
end
def next_boundary(time)
case @grain
when 'day' then time.beginning_of_day + 1.day
when 'month' then time.beginning_of_month.next_month
else time.beginning_of_year.next_year
end
end
def periods
@before = timestamp('before') || @now.beginning_of_month
@from = timestamp('from') || @before.prev_year
raise ArgumentError if @from >= @before || @before > @now
rows = []
boundary = next_boundary(@from)
while boundary <= @before
raise ArgumentError if rows.length >= MAX_PERIODS
format = { 'day' => '%Y-%m-%d', 'month' => '%Y-%m', 'year' => '%Y' }.fetch(@grain)
rows << [boundary, (boundary - 1.second).strftime(format)]
boundary = next_boundary(boundary)
end
rows
end
def trend
points = periods
metadata = { count: points.length, group_count: points.length, state_rows: [],
from: @from.iso8601, before: @before.iso8601, grain: @grain }
return metadata if points.empty?
values = points.map do |boundary, period|
"SELECT CAST(#{ quote(boundary) } AS DATETIME(6)) AS boundary, " +
"#{ quote(period) } AS period"
end.join(' UNION ALL ')
order = order_sql({ 'period' => 'boundary', 'count' => 'post_count' },
'period:asc', ['boundary ASC'])
known = @post_start ? "periods.boundary > #{ quote(@post_start) }" : 'FALSE'
rows = @connection.select_all(<<~SQL)
WITH periods AS (#{ values }), series AS (
SELECT periods.boundary, periods.period,
CASE WHEN #{ known } THEN COUNT(DISTINCT p.post_id) ELSE NULL END AS post_count
FROM periods
LEFT JOIN post_versions p ON p.event_type <> 'discard'
AND #{ membership('p') }
AND #{ latest_condition('post_versions', 'p', 'periods.boundary') }
AND #{ tag_visible('periods.boundary') }
GROUP BY periods.boundary, periods.period
)
SELECT * FROM series ORDER BY #{ order }
LIMIT #{ @limit } OFFSET #{ (@page - 1) * @limit }
SQL
metadata.merge(state_rows: rows.map do |row|
{ kind: @kind, tag_id: @tag, name: nil, period: row['period'],
at: row['boundary'].utc.iso8601(6), count: row['post_count']&.to_i }
end)
end
def order_sql(expressions, default, final)
raw = @params.fetch('sort', default)
sorts = raw.empty? ? [] : raw.split(',', -1).map { _1.split(':', -1) }
raise ArgumentError if sorts.map(&:first).uniq.length != sorts.length
raise ArgumentError if sorts.length > expressions.length
order = sorts.flat_map do |key, direction, *extra|
if extra.any? || !expressions.key?(key) || !['asc', 'desc'].include?(direction)
raise ArgumentError
end
expression = expressions.fetch(key)
["(#{ expression }) IS NULL ASC", "#{ expression } #{ direction.upcase }"]
end
(order + final).join(', ')
end
end
+172
ファイルの表示
@@ -0,0 +1,172 @@
require 'csv'
# Export uses the exact same population and grouped cells as the interactive table.
class CountsTableExport
EXTRA = ['format', 'columns', 'headers', 'labels', 'children', 'child_sort',
'order', 'nested_json', 'summary_columns', 'basis', 'axis', 'latest'].freeze
attr_reader :format, :resource
def initialize(params)
raise ArgumentError if (params.keys - CountsTableQuery::KEYS - EXTRA).any?
raise ArgumentError unless params.values.all? { _1.is_a?(String) }
@params = params
@labels = JSON.parse(params.fetch('labels', '{}'))
raise ArgumentError unless @labels.is_a?(Hash)
@selected = params.key?('children') ? params['children'].split(',') : nil
@order = params.fetch('order', '').split(',')
@child_sort = params.fetch('child_sort', '').split(',').map { _1.split(':', -1) }
@child_sort.each do |path, direction, *extra|
raise ArgumentError if extra.any? || !['asc', 'desc'].include?(direction)
raise ArgumentError unless path&.match?(/\A[a-zA-Z_]+(?:\.[a-zA-Z]+)*\/[a-zA-Z]+\z/)
end
@format = params.fetch('format', 'json')
@resource = params.fetch('resource', 'posts')
raise ArgumentError unless ['json', 'csv'].include?(format)
@query = CountsTableQuery.new(params.slice(*CountsTableQuery::KEYS)
.merge('page' => '1', 'limit' => '100'))
@summary = @query.mode == 'summary'
@columns = params.fetch('columns', '').split(',')
@headers = JSON.parse(params.fetch('headers', '[]'))
raise ArgumentError unless @headers.is_a?(Array) && @headers.all? { _1.is_a?(String) }
unless @summary
if ['posts', 'tags'].include?(resource)
@projection = CountsExportProjection.new(params, resource:, summary: false, tags: false)
else
allowed = @query.population.definition.fields.keys + ['is_latest']
raise ArgumentError if (@columns - allowed).any?
raise ArgumentError unless @columns.length == @headers.length && @columns.uniq == @columns
end
end
end
def body
Enumerator.new do |output|
ApplicationRecord.transaction(isolation: :repeatable_read) do
@language_names = Language.pluck(:code, :name).to_h
started = false
first = true
@query.each_page do |page|
unless started
metadata = page.except(:versions, :cells, :axis_labels, :column_values).merge(
resource:, filters: @params.slice(*CountsTableQuery::KEYS),
fields: @query.population.definition.metadata,
columns: @columns, column_labels: @headers,
order: @params['order'], children: @params['children'],
sort: @params.fetch('sort', ''), child_sort: @params['child_sort'])
output << (format == 'json' ? "{\"metadata\":#{ metadata.to_json },\"data\":[" : "\uFEFF")
if format == 'csv'
headers = @summary ? summary_headers : @headers
output << CSV.generate_line(headers.map { safe(_1) }, row_sep: "\r\n")
end
started = true
end
(page[:cells] || page[:versions]).each do |row|
record = project(row)
if format == 'json'
output << ',' unless first
output << record.to_json
else
values = if @summary
row[:values].each_with_index.map do |value, index|
page[:axis_labels].fetch("a#{ index }", {}).fetch(value.to_s, value)
end + [row[:count]]
elsif @projection
@projection.csv_record(record, row)
else
record.map { |key, value| csv_logical(key, value) }
end
output << CSV.generate_line(values.map { safe(_1) }, row_sep: "\r\n")
end
first = false
end
end
output << ']}' if format == 'json'
end
end
end
private
def summary_headers
labels = @labels
@query.axes.all.map { labels.fetch(_1['field'], _1['field']) } +
[labels.fetch('count', 'count')]
end
def project(row)
return { row: row[:values].first(@query.axes.rows.length),
column: row[:values].drop(@query.axes.rows.length), count: row[:count] } if @summary
return @projection.project(row) if @projection
@columns.to_h do |column|
key = column == 'user' ? 'created_by_user' : column
value = row[key]
value = project_nested(value, key) if ['members', 'placements', 'export_paths_json'].include?(key)
[key, value]
end
end
def camel(key)
key.gsub(/_([a-z])/) { Regexp.last_match(1).upcase }
end
def project_nested(value, path)
rows = path == 'export_paths_json' ? (value || {}).map { |profile, target|
{ 'profile' => profile, 'path' => target }
} : Array(value)
sorts = @child_sort.filter_map do |key, direction|
parent, field = key.split('/', 2)
[field, direction] if parent == path
end
rows = rows.each_with_index.sort do |(first, a), (last, b)|
compared = 0
sorts.each do |field, direction|
left = first.find { |key, _| camel(key) == field }&.last
right = last.find { |key, _| camel(key) == field }&.last
compared = if left.nil? || right.nil?
left.nil? == right.nil? ? 0 : (left.nil? ? 1 : -1)
else
(left.to_s <=> right.to_s) * (direction == 'asc' ? 1 : -1)
end
break unless compared.zero?
end
compared.zero? ? a <=> b : compared
end.map(&:first)
rows.map do |row|
fields = row.keys.select { !@selected || @selected.include?("#{ path }.#{ camel(_1) }") }
ordered = (@order + fields.map { "#{ path }.#{ camel(_1) }" }).uniq
.filter_map { |item| fields.find { "#{ path }.#{ camel(_1) }" == item } }
ordered.to_h do |key|
item = row[key]
item = project_nested(item, "#{ path }.#{ camel(key) }") if item.is_a?(Array)
[key, item]
end
end
end
def csv_logical(key, value)
return value ? @labels.fetch('latest', '') : '' if key == 'is_latest'
if value.is_a?(Time) || value.is_a?(ActiveSupport::TimeWithZone)
return @query.population.timezone.utc_to_local(value.utc).iso8601
end
if key == 'language_code' || key == 'source_language_code'
return @language_names[value]
end
if value.is_a?(Hash)
return value.to_h { |field, item| [@labels.fetch(camel(field.to_s), ''), csv_logical(field, item)] }
end
return value.map { csv_logical(key, _1) } if value.is_a?(Array)
@labels.fetch("#{ key }:#{ value }", value)
end
def safe(value)
return value if value.is_a?(Numeric)
text = value.is_a?(Hash) || value.is_a?(Array) ? JSON.generate(value) : value.to_s
text.match?(/\A\s*[=+@-]/) ? "'#{ text }" : text
end
end
+252
ファイルの表示
@@ -0,0 +1,252 @@
# Common entrypoint for six version resources; legacy endpoints keep their contracts.
class CountsTableQuery
MAX_COLUMNS = 100
class TooManyColumns < ArgumentError; end
KEYS = ['resource', 'definition', 'mode', 'at', 'execution_at', 'from', 'through',
'timezone', 'rows', 'cols', 'conditions', 'drill', 'tags', 'tag_refs', 'match',
'tag', 'kind', 'resource_id', 'user_id', 'page', 'limit', 'sort', 'locale'].freeze
attr_reader :population, :axes, :page, :limit, :mode
def initialize(params)
raise ArgumentError if (params.keys - KEYS).any?
raise ArgumentError unless params.values.all? { _1.is_a?(String) }
@params = params
@population = CountsPopulation.new(params)
@definition = population.definition
@connection = @definition.model.connection
@axes = CountsAxes.new(population)
@page = Integer(params.fetch('page', '1'), 10)
@limit = Integer(params.fetch('limit', '20'), 10)
@mode = params.fetch('mode', 'summary')
raise ArgumentError unless page.between?(1, 1_000_000) && limit.between?(1, 100)
raise ArgumentError unless ['summary', 'details'].include?(mode)
@sort = params.fetch('sort', '').split(',').map { _1.split(':', -1) }
allowed = mode == 'summary' ? axes.all.each_index.map { "a#{ _1 }" } + ['count'] :
@definition.fields.keys + ['is_latest'] +
(@definition.resource == 'posts' ? ['tags', 'parents', 'original_created_at'] : []) +
(@definition.resource == 'tags' ? ['parents', 'deprecated'] : [])
raise ArgumentError if @sort.map(&:first).uniq.length != @sort.length
@sort.each do |key, direction, *extra|
measure = mode == 'summary' && key&.match?(/\Ameasure_[0-9a-f]+\z/)
measure_values(key) if measure
if extra.any? || (!allowed.include?(key) && !measure) || !['asc', 'desc'].include?(direction)
raise ArgumentError
end
end
end
def call
ApplicationRecord.transaction do
metadata.merge(mode == 'summary' ? summary : details)
end
end
def metadata
{ page:, limit:, execution_at: population.at.iso8601(6), at: @params['at'].presence,
definition: population.history ? 'history' : 'latest',
timezone: population.timezone.identifier, rows: axes.rows, cols: axes.columns,
fields: @definition.metadata,
count: population.scope.except(:select).optimizer_hints('MAX_EXECUTION_TIME(10000)').count }
end
def summary
expressions = axes.expressions
selected = expressions.each_with_index.map do |expression, index|
"#{ expression } AS a#{ index }"
end
source = population.scope
source = source.joins(axes.joins) unless axes.joins.empty?
sql = if expressions.empty?
source.except(:select).select('COUNT(DISTINCT v.id) AS count').to_sql
else
source.except(:select).group(*expressions.map { Arel.sql(_1) })
.select(Arel.sql((selected + ['COUNT(DISTINCT v.id) AS count']).join(', ')))
.to_sql
end
prefix = "WITH cells AS (#{ sql })"
column_values = limited_columns(prefix)
row_keys = axes.rows.each_index.map { "a#{ _1 }" }
# Page whole row tuples so a pivot row is never split across display pages.
row_selection = row_keys.empty? ? '1 AS singleton' : row_keys.join(', ')
row_group = row_keys.empty? ? 'GROUP BY singleton' : "GROUP BY #{ row_keys.join(', ') }"
row_sort = @sort.each_with_index.filter_map do |(key, direction), index|
if key == 'count' || key.start_with?('measure_')
condition = if key == 'count'
'TRUE'
else
measure_values(key).each_with_index.map do |value, offset|
"a#{ axes.rows.length + offset } <=> #{ @connection.quote(value) }"
end.join(' AND ').presence || 'TRUE'
end
["MAX(CASE WHEN #{ condition } THEN count END) AS sort#{ index }",
"sort#{ index } IS NULL ASC, sort#{ index } #{ direction.upcase }"]
elsif row_keys.include?(key)
[nil, "#{ key } IS NULL ASC, #{ key } #{ direction.upcase }"]
end
end
extras = row_sort.filter_map(&:first)
order_rows = row_sort.map(&:last) + row_keys.map { "#{ _1 } IS NULL, #{ _1 }" }
order_rows = ['singleton ASC'] if order_rows.empty?
group_sql = "SELECT #{ ([row_selection] + extras).join(', ') } FROM cells #{ row_group }"
total = @connection.select_value(<<~SQL).to_i
#{ prefix }
SELECT /*+ MAX_EXECUTION_TIME(10000) */ COUNT(*) FROM (#{ group_sql }) row_groups
SQL
join = row_keys.map { "cells.#{ _1 } <=> page_rows.#{ _1 }" }.join(' AND ')
join = 'TRUE' if row_keys.empty?
order_cols = @sort.filter_map do |key, direction|
next unless key.match?(/\Aa[0-9]+\z/) && !row_keys.include?(key)
"cells.#{ key } #{ direction.upcase }"
end
order = ['page_rows.position'] + order_cols +
axes.all.each_index.map { "cells.a#{ _1 }" }
records = @connection.select_all(<<~SQL)
#{ prefix }, row_groups AS (#{ group_sql }),
numbered AS (SELECT row_groups.*,
ROW_NUMBER() OVER (ORDER BY #{ order_rows.join(', ') }) AS position FROM row_groups),
page_rows AS (SELECT * FROM numbered
WHERE position > #{ (page - 1) * limit } AND position <= #{ page * limit })
SELECT /*+ MAX_EXECUTION_TIME(10000) */ cells.* FROM cells JOIN page_rows ON #{ join }
ORDER BY #{ order.join(', ') }
SQL
{ group_count: total, column_values:, cells: records.map do |row|
{ values: axes.all.each_index.map { row["a#{ _1 }"] }, count: row['count'].to_i }
end, axis_labels: axis_labels(records, column_values) }
end
def details
query = population.scope.select(Arel.sql(
"(#{ population.latest_sql }) AS counts_latest"))
order = sort_sql { detail_expression(_1) }
query = query.order(Arel.sql((order + ['v.id DESC']).join(', ')))
.limit(limit).offset((page - 1) * limit)
rows = query.to_a
if ['posts', 'tags'].include?(@definition.resource)
point = population.history ? population.at :
population.timestamp(@params['at']) || population.at
values = CountsVersionRepr.many(rows, resource: @definition.resource,
language_code: locale.language_code,
at: point)
values.each_with_index { |value, index| value[:event_type] = rows[index].event_type }
else
users = User.where(id: rows.map(&:created_by_user_id)).pluck(:id, :name).to_h
values = rows.map do |row|
attributes = @definition.fields.keys.to_h do |key|
[key, row.public_send(@definition.column(key))]
end
attributes.merge('id' => row.id, 'resource_id' => row[@definition.keys.first],
'is_latest' => row.attributes['counts_latest'].to_i == 1,
'created_by_user' => row.created_by_user_id &&
{ id: row.created_by_user_id, name: users[row.created_by_user_id] })
end
end
{ versions: values }
end
def each_page
total = nil
loop do
result = call
total ||= mode == 'summary' ? result[:group_count] : result[:count]
yield result
break if page * limit >= total
@page += 1
end
end
private
def locale = @locale ||= Locale.find_by!(code: @params.fetch('locale', 'ja'))
def limited_columns(prefix)
keys = axes.columns.each_index.map { "a#{ axes.rows.length + _1 }" }
return [[]] if keys.empty?
order = keys.map { "#{ _1 } IS NULL, #{ _1 }" }.join(', ')
values = @connection.select_rows(<<~SQL)
#{ prefix }
SELECT /*+ MAX_EXECUTION_TIME(10000) */ DISTINCT #{ keys.join(', ') } FROM cells
ORDER BY #{ order } LIMIT #{ MAX_COLUMNS + 1 }
SQL
raise TooManyColumns if values.length > MAX_COLUMNS
values
end
def measure_values(key)
value = JSON.parse([key.delete_prefix('measure_')].pack('H*'))
unless (value.is_a?(Array) && value.length == axes.columns.length &&
value.all? { _1.nil? || _1.is_a?(String) || _1.is_a?(Numeric) })
raise ArgumentError
end
value
end
def detail_expression(key)
return "(#{ population.latest_sql })" if key == 'is_latest'
aliases = { 'tags' => 'tags_json', 'parents' => @definition.resource == 'posts' ?
'parent_post_ids' : 'parent_tag_ids', 'deprecated' => 'deprecated_at',
'original_created_at' => 'original_created_from' }
field = aliases.fetch(key, key)
expression = @definition.expression(field)
return expression unless @definition.fields[field] == :json
"CAST(#{ expression } AS CHAR) COLLATE utf8mb4_bin"
end
def sort_sql
@sort.flat_map do |key, direction|
expression = yield key
["(#{ expression }) IS NULL ASC", "#{ expression } #{ direction.upcase }"]
end
end
def axis_labels(records, column_values)
result = {}
axes.all.each_with_index do |axis, index|
values = records.map { _1["a#{ index }"] }
if index >= axes.rows.length
values += column_values.map { _1[index - axes.rows.length] }
end
values = values.compact.uniq
labels = case axis['field']
when 'user'
User.where(id: values).pluck(:id, :name).to_h.transform_keys(&:to_s)
when 'language_code', 'source_language_code'
Language.where(code: values).pluck(:code, :name).to_h
when 'tags'
tag_labels(values)
else {}
end
result["a#{ index }"] = labels
end
result
end
def tag_labels(values)
internal = values.grep(/\Ainternal:/).map { _1.split(':').last.to_i }
external = values.grep(/\Aexternal:/).map { _1.split(':').last.to_i }
point = population.timestamp(@params['at']) || population.at
labels = {}
sources = [[TagVersion, internal, 'internal'], [NicoTagVersion, external, 'external']]
sources.each do |model, ids, kind|
next if ids.empty?
table = model.table_name
model.where(tag_id: ids).where('created_at <= ?', point)
.where("NOT EXISTS (SELECT 1 FROM #{ table } newer " +
"WHERE newer.tag_id = #{ table }.tag_id AND newer.created_at <= ? " +
"AND newer.version_no > #{ table }.version_no)", point).each do |row|
names = row.respond_to?(:names) ? row.names : nil
name = names&.dig(locale.language_code, 'name') || row.name
labels["#{ kind }:#{ row.tag_id }"] = name
end
end
labels
end
end
+12 -15
ファイルの表示
@@ -95,14 +95,10 @@ class PostCreatePlan
def build_default_tag_specs locale, direct_tag_specs
default_tag_specs = []
if direct_tag_specs.length < 10 && direct_tag_specs.none? { _1[:name] == TAGME_TAG_NAME }
default_tag_specs << {
name: TAGME_TAG_NAME,
category: :meta }
default_tag_specs << { name: TAGME_TAG_NAME, category: :meta }
end
if direct_tag_specs.none? { deerjikist_tag_spec?(locale, _1) }
default_tag_specs << {
name: NO_DEERJIKIST_TAG_NAME,
category: :meta }
default_tag_specs << { name: NO_DEERJIKIST_TAG_NAME, category: :meta }
end
default_tag_specs
@@ -117,9 +113,9 @@ class PostCreatePlan
end
def validate_new_tag_spec! locale, spec
tag_name = TagName.new(name: spec[:name], language_code: locale.language_code,
script_code: locale.script_code, primary_flg: true)
tag = Tag.new(category: spec[:category])
tag_name = TagName.new(tag:, name: spec[:name], language_code: locale.language_code,
script_code: locale.script_code, primary_flg: true)
return if tag_name.valid? && tag.valid?
post = Post.new
@@ -140,10 +136,10 @@ class PostCreatePlan
end
expanded_parent_specs =
Tag.expand_parent_tags(existing_snapshot_tags)
.reject(&:deprecated?)
.map { |tag|
{ name: tag.name(locale.language_code),
category: tag.category.to_sym }
.reject(&:deprecated?)
.map { |tag|
{ name: tag.name(locale.language_code),
category: tag.category.to_sym }
}
merge_tag_specs(snapshot_tag_specs + expanded_parent_specs)
end
@@ -164,9 +160,10 @@ class PostCreatePlan
def existing_tags_by_name locale
@existing_tags_by_name ||= begin
names = tag_names.map { canonical_tag_name_without_sections(locale, _1) }.uniq
TagName.where(language_code: locale.language_code, primary_flg: true, name: names)
.includes(:tag)
.each_with_object({ }) { |tag_name, tags| tags[tag_name.name] = tag_name.tag }
TagName
.where(language_code: locale.language_code, primary_flg: true, name: names)
.includes(:tag)
.each_with_object({ }) { |tag_name, tags| tags[tag_name.name] = tag_name.tag }
end
end
+1 -1
ファイルの表示
@@ -105,7 +105,7 @@ class PostCreator
category = spec[:category] || spec['category']
next if name.blank? || category.blank?
tag = Tag.find_or_create_by_tag_name!(locale, name, category:)
tag = Tag.find_or_create_by_tag_name!(locale, name, category:, created_by_user: @actor)
tag.update!(category:) if tag.category.to_sym != category.to_sym
tags[name] ||= tag
end.values
+42
ファイルの表示
@@ -0,0 +1,42 @@
# Shared literal parsing and reference resolution for current and snapshot post search.
class PostTagSearch
def self.parse_references(raw, literals)
return nil if raw.blank?
references = JSON.parse(raw)
valid = references.is_a?(Array) && references.all? do |reference|
next false unless reference.is_a?(Hash) && reference['name'].is_a?(String)
ids = [reference['tag_id'], reference['external_tag_id']].compact
((ids.any? || reference['missing'] == true) &&
ids.all? { _1.is_a?(Integer) && _1.positive? } &&
(!reference.key?('missing') || (reference['missing'] == true && ids.empty?)) &&
(!reference.key?('negative') || [true, false].include?(reference['negative'])))
end
raise ArgumentError unless valid
references.map { _1['name'] } == literals ? references : nil
end
def self.literals(locale, raw, references: nil)
raw.split.each_with_index.filter_map do |token, index|
next if token.downcase == 'type:or'
reference = references&.[](index)
name = token.sub(/\Anot:/i, '')
name = TagName.canonicalise(locale, name).first unless reference
negative = reference&.key?('negative') ? reference['negative'] :
token.downcase.start_with?('not:')
{ name:, negative:, reference: }
end
end
def self.references(locale, literal)
reference = literal[:reference]
return [reference['tag_id'], reference['external_tag_id']] if reference
[TagName.where(language_code: locale.language_code, name: literal[:name])
.distinct.pluck(:tag_id),
ExternalTag.where("CONCAT(platform, ':', name) = ?", literal[:name]).pluck(:id)]
end
end
+137
ファイルの表示
@@ -0,0 +1,137 @@
require 'digest'
class SharedResourceIdAllocator
SPACES = {
'tags' => ['tags', 'external_tags'],
'wiki' => ['tag_descriptions', 'tag_groups'],
'wiki_history' => ['tag_description_versions', 'tag_group_versions'] }.freeze
HISTORY_IDS = {
'tags' => [
['tag_versions', 'tag_id'],
['nico_tag_versions', 'tag_id'],
['tag_description_versions', 'tag_id']],
'wiki' => [['tag_group_versions', 'tag_group_id']] }.freeze
LOCK_TIMEOUT = 10
CONNECTION_LOCKS = ObjectSpace::WeakMap.new
class LockCleanup
def initialize &finish
@finish = finish
end
def trigger_transactional_callbacks? = false
def before_committed! = nil
def committed!(**) = @finish.call(self)
def rolledback!(**) = @finish.call(self)
end
private_constant :LockCleanup
def self.assign! record
space, tables = SPACES.find { |_name, names| names.include?(record.class.table_name) }
raise ArgumentError, 'Unknown shared ID resource' unless tables
connection = record.class.connection
lock_space!(record.class, space)
# 明示-tic な Id. 指定は許可する.
return if record.id.present?
next_id = record.class.uncached do
historical_max = HISTORY_IDS.fetch(space, []).map { |table, column|
quoted_table = connection.quote_table_name(table)
quoted_column = connection.quote_column_name(column)
connection.select_value("SELECT MAX(#{ quoted_column }) FROM #{ quoted_table }").to_i
}.max.to_i
[next_ids(connection, tables).max, historical_max + 1].max
end
record.id = next_id
end
def self.lock_space! record_class, space
raise ArgumentError, 'Unknown shared ID resource' unless SPACES.key?(space)
connection = record_class.connection
transaction = connection.current_transaction
unless transaction.open?
raise ActiveRecord::ActiveRecordError, 'Shared ID requires a transaction'
end
database = connection.pool.db_config.database
name = "btrc:ids:#{ Digest::SHA256.hexdigest(database)[0, 16] }:#{ space }"
held_locks = CONNECTION_LOCKS[connection] ||= {}
return if held_locks.key?(name)
quoted_name = connection.quote(name)
acquired = record_class.uncached do
connection.select_value("SELECT GET_LOCK(#{ quoted_name }, #{ LOCK_TIMEOUT })")
end
unless acquired.to_i == 1
raise ActiveRecord::LockWaitTimeout, "Shared ID lock unavailable: #{ space }"
end
held_locks[name] = true
finish = lambda do |cleanup|
next unless held_locks.key?(name)
# Savepoint callbacks may run before the outer transaction ends.
# Keep one session lock and transfer its cleanup to the remaining parent.
parent = connection.current_transaction
if parent.open?
connection.add_transaction_record(cleanup)
next
end
result = record_class.uncached do
connection.select_value("SELECT RELEASE_LOCK(#{ quoted_name })")
end
unless result.to_i == 1
raise ActiveRecord::ActiveRecordError, "Shared ID lock release failed: #{ space }"
end
held_locks.delete(name)
rescue StandardError
# Never return a session with an untracked advisory lock to the pool.
begin
connection.disconnect!
ensure
held_locks.clear
end
raise
end
begin
# Rails finalises remaining records in ensure if another callback raises.
connection.add_transaction_record(LockCleanup.new(&finish))
rescue StandardError
begin
connection.disconnect!
ensure
held_locks.clear
end
raise
end
end
def self.next_ids connection, tables
expiry = connection.select_value('SELECT @@SESSION.information_schema_stats_expiry').to_i
begin
connection.execute('SET SESSION information_schema_stats_expiry = 0')
quoted_tables = tables.map { connection.quote(_1) }.join(', ')
rows = connection.select_all(<<~SQL).to_a
SELECT TABLE_NAME, ENGINE, AUTO_INCREMENT
FROM information_schema.tables
WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME IN (#{ quoted_tables })
SQL
persistent = rows.all? do |row|
row['ENGINE'] == 'InnoDB' && row['AUTO_INCREMENT'].to_i.positive?
end
unless rows.size == tables.size && persistent
raise ActiveRecord::ActiveRecordError, 'Persistent InnoDB counters are required'
end
rows.map { _1.fetch('AUTO_INCREMENT').to_i }
ensure
connection.execute("SET SESSION information_schema_stats_expiry = #{ expiry }")
end
end
private_class_method :next_ids
end
+49 -34
ファイルの表示
@@ -1,54 +1,69 @@
class TagDescriptionOperation
LANGUAGE_CODE = 'ja'
class Conflict < StandardError
;
end
def self.create!(tag:, body:, created_by_user:, message: nil)
def self.create! locale, tag:, body:, created_by_user:, message: nil
ApplicationRecord.transaction do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
tag.lock!
raise ActiveRecord::RecordNotUnique if TagDescription.exists?(
tag_id: tag.id, language_code: LANGUAGE_CODE)
tag_id: tag.id, language_code: locale.language_code)
page = Wiki::CompatibilityWriter.create!(
tag_name: tag.tag_name(LANGUAGE_CODE), body:, created_by_user:, message:)
revision = page.wiki_revisions.order(id: :desc).first
record_version!(tag:, revision:, body:, user: created_by_user,
description: nil)
page
description = TagDescription.new(tag:, language_code: locale.language_code)
body = normalise_body!(description, body)
record_version!(description:, body:, user: created_by_user)
description
end
end
def self.update!(tag:, body:, updated_by_user:, message: nil,
base_revision_id: nil)
def self.update! locale, tag:, body:, updated_by_user:, message: nil, base_revision_id: nil
ApplicationRecord.transaction do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
tag.lock!
description = TagDescription.lock.find_by!(
tag_id: tag.id, language_code: LANGUAGE_CODE)
page = WikiPage.lock.find(description.id)
revision = Wiki::CompatibilityWriter.update!(
page:, body:, updated_by_user:, message:,
base_revision_id:)
next unless revision
tag_id: tag.id, language_code: locale.language_code)
body = normalise_body!(description, body)
latest = TagDescriptionVersion.find_by(
tag_id: tag.id, language_code: locale.language_code,
version_no: description.version_no)
if base_revision_id.present? && latest&.id != base_revision_id.to_i
raise Conflict, '競合が発生してゐます.'
end
next if latest && normalise_body(description.body) == body
record_version!(tag:, revision:, body: revision.body,
user: updated_by_user, description:)
revision
record_version!(description:, body:, user: updated_by_user)
end
end
def self.record_version! tag:, revision:, body:, user:, description:
version_no = description ? description.version_no + 1 : 1
def self.record_version! description:, body:, user:
creating = description.new_record?
version_no = creating ? 1 : description.version_no + 1
description.created_by_user_id = user.id if creating
description.update!(body:, version_no:, updated_by_user_id: user.id)
TagDescriptionVersion.create!(
id: revision.id, tag_id: tag.id, language_code: LANGUAGE_CODE,
version_no:, event_type: description ? 'update' : 'create', body:,
created_by_user_id: user.id, created_at: revision.created_at)
if description
description.update!(body:, version_no:, updated_by_user_id: user.id)
else
TagDescription.create!(
id: revision.wiki_page_id, tag_id: tag.id,
language_code: LANGUAGE_CODE, version_no:, body:,
created_by_user_id: user.id, updated_by_user_id: user.id)
end
tag_id: description.tag_id, language_code: description.language_code,
version_no:, event_type: creating ? 'create' : 'update', body:,
source_language_code: description.source_language_code,
source_version_no: description.source_version_no,
created_by_user_id: user.id, created_at: Time.current)
end
private_class_method :record_version!
def self.normalise_body body
body.to_s.gsub(/\r\n?/, "\n")
.encode('UTF-8', invalid: :replace, undef: :replace, replace: '🖕')
.sub(/\n+\z/, '')
end
private_class_method :normalise_body
def self.normalise_body! description, body
normalised = normalise_body(body)
return normalised if normalised.present?
description.errors.add(:body, :blank)
raise ActiveRecord::RecordInvalid, description
end
private_class_method :normalise_body!
end
+178
ファイルの表示
@@ -0,0 +1,178 @@
require 'kramdown'
class TagGroupMarkdownParser
PREFIX = TagGroupRepr::PREFIX
class InvalidDocument < StandardError
;
end
def self.parse(locale, body:, group: nil)
new(body:, group:).parse(locale)
end
def initialize(body:, group:)
@body = body.to_s.gsub(/\r\n?/, "\n")
@group = group
end
def parse locale
parse_body = @body.gsub(/([^\n])\n([\x23]{1,6}[ \t]+\S)/, "\\1\n\n\\2")
root = Kramdown::Document.new(parse_body, input: 'GFM', auto_ids: false).root
placements = []
description_parts = []
headings = []
root.children.each do |node|
case node.type
when :blank
next
when :header
level = node.options.fetch(:level)
name = inline_text(node).strip
invalid!('Section 名は必須です.') if name.blank?
headings.pop while headings.any? && headings.last.fetch(:level) >= level
parent = headings.last&.fetch(:node)
section = { 'section_name' => name, 'description' => nil,
'children' => [] }
(parent ? parent.fetch('children') : placements) << section
headings << { level:, node: section }
when :p
text = inline_markdown(node).strip
next if text.empty?
if headings.any?
section = headings.last.fetch(:node)
section['description'] = [section['description'], text].compact.join("\n\n")
else
description_parts << text
end
when :ul
invalid!('未対応のリスト項目があります.') if node.attr.any?
target = headings.any? ? headings.last.fetch(:node).fetch('children') : placements
target.concat(node.children.map { parse_list_item(locale, _1) })
else
invalid!("未対応の Markdown 構文 #{ node.type.inspect } です.")
end
end
description = description_parts.presence&.join("\n\n")
{ description:, placements:, members: build_members(placements),
compatibility_body: @body.sub(/\n+\z/, '') }
rescue Kramdown::Error => error
invalid!("Markdown を解析できません:#{ error.message }")
end
private
def parse_list_item locale, item
invalid!('リスト項目ではありません.') unless item.type == :li
invalid!('未対応のリスト項目があります.') if item.attr.any?
nested, own = item.children.partition { |child| child.type == :ul }
invalid!('順序付きリストは使用できません.') if own.any? { _1.type == :ol }
own.reject! { _1.type == :blank }
invalid!('未対応のリスト項目があります.') unless own.all? do |child|
[:p, :text].include?(child.type)
end
invalid!('リスト項目は一つの段落で指定してください.') if own.size != 1
label = own.map { inline_text(_1) }.join(' ').strip
invalid!('空のリスト項目は使用できません.') if label.blank?
node =
if (group_name = TagGroupRepr.name_from_title(label))
group_name = group_name.strip
child_group = TagGroup.find_by(name: group_name)
invalid!("Tag Group #{ group_name.inspect } を解決できません.") unless child_group
invalid!('自分自身を含めることはできません.') if child_group.id == @group&.id
{ 'tag_group_id' => child_group.id }
elsif label.end_with?(':', ':')
section_name = label.sub(/[::]\z/, '').strip
invalid!('Section 名は必須です.') if section_name.blank?
{ 'section_name' => section_name, 'description' => nil }
else
{ 'tag_id' => resolve_tag(locale, label) }
end
node['children'] = nested.flat_map do |list|
invalid!('未対応の入れ子リストです.') unless list.type == :ul
list.children.map { parse_list_item(locale, _1) }
end
node
end
def resolve_tag locale, name
names = TagName.where(language_code: locale.language_code, name:).to_a
tag_ids = names.filter_map(&:tag_id).uniq
invalid!("Tag #{ name.inspect } を解決できません.") if tag_ids.empty?
invalid!("Tag 名 #{ name.inspect } が曖昧です.") if tag_ids.many?
tag_ids.first
end
def build_members(nodes)
references = []
visit = lambda do |items|
items.each do |node|
references << node.slice('tag_id', 'tag_group_id') if
node.key?('tag_id') || node.key?('tag_group_id')
visit.call(node.fetch('children'))
end
end
visit.call(nodes)
references.uniq
end
def inline_text(node)
if node.attr.any?
invalid!("未対応のインライン構文 #{ node.type.inspect } です.")
end
case node.type
when :text
node.value.to_s
when :p, :header
node.children.map { inline_text(_1) }.join
else
invalid!("未対応のインライン構文 #{ node.type.inspect } です.")
end
end
def inline_markdown(node)
case node.type
when :text
node.value.to_s
when :codespan
text = node.value.to_s
longest = text.scan(/`+/).map(&:length).max || 0
delimiter = '`' * (longest + 1)
padded = text.start_with?('`', ' ') || text.end_with?('`', ' ')
content = padded ? " #{ text } " : text
"#{ delimiter }#{ content }#{ delimiter }"
when :softbreak
"\n"
when :br
" \n"
when :p
node.children.map { inline_markdown(_1) }.join
when :a
href = node.attr.fetch('href')
label = node.children.map { inline_markdown(_1) }.join
title = node.attr['title']&.gsub('"', '\\"')
suffix = title ? " \"#{ title }\"" : ''
"[#{ label }](#{ href }#{ suffix })"
when :em
"*#{ node.children.map { inline_markdown(_1) }.join }*"
when :strong
"**#{ node.children.map { inline_markdown(_1) }.join }**"
else
invalid!("未対応の説明文構文 #{ node.type.inspect } です.")
end
end
def invalid! message
raise InvalidDocument, message
end
end
+123
ファイルの表示
@@ -0,0 +1,123 @@
class TagGroupMarkdownRenderer
class UnsupportedStructure < StandardError
;
end
def self.render group
new(group).render
end
def initialize group
@group = group
end
def render
placements = @group.fetch('placements')
validate_membership!(placements)
blocks = []
if @group['description'].present?
blocks << clean_text(@group['description'])
end
first_section = placements.index { _1.key?('section_name') }
if first_section
if placements.drop(first_section).any? { !_1.key?('section_name') }
raise UnsupportedStructure,
'見出し Section の後に最上位 Placement は配置できません.'
end
if first_section.positive?
blocks << render_list(placements.take(first_section), 0)
end
blocks.concat(placements.drop(first_section).map { render_heading(_1, 2) })
elsif placements.any?
blocks << render_list(placements, 0)
end
blocks.compact.reject(&:empty?).join("\n").sub(/\n+\z/, '')
end
private
def validate_membership! placements
placed = placement_members(placements).uniq.sort
members = @group.fetch('members').map { |member|
if member['tag']
['tag', member.fetch('tag').fetch('id')]
else
['tag_group', member.fetch('tag_group').fetch('id')]
end
}.uniq.sort
missing = members - placed
if missing.any?
raise UnsupportedStructure, '掲載位置のない membership は Markdown にできません.'
end
end
def placement_members nodes
nodes.flat_map do |node|
own = if node['tag']
[['tag', node.fetch('tag').fetch('id')]]
elsif node['tag_group']
[['tag_group', node.fetch('tag_group').fetch('id')]]
else
[]
end
own + placement_members(node.fetch('children'))
end
end
def render_heading section, level
header = "#{ '#' * level } #{ clean_text(section.fetch('section_name')) }"
blocks = [header]
blocks << clean_text(section['description']) if section['description'].present?
children = section.fetch('children')
section_indexes = children.each_index.select { |index| children[index].key?('section_name') }
if section_indexes.any?
first_section = section_indexes.first
if children[first_section..].any? { !_1.key?('section_name') }
raise UnsupportedStructure,
'見出し Section と兄弟 Placement の交互配置は Markdown にできません.'
end
list_nodes = children[0...first_section]
blocks << render_list(list_nodes, 0) if list_nodes.any?
blocks.concat(children[first_section..].map { render_heading(_1, level + 1) })
elsif children.any?
blocks << render_list(children, 0)
end
blocks.join("\n")
end
def render_list nodes, depth
nodes.map do |node|
indent = ' ' * depth
if node.key?('section_name')
raise UnsupportedStructure,
'リスト内 Section の説明文は Markdown にできません.' if node['description'].present?
children = node.fetch('children')
head = "#{ indent }- #{ clean_text(node.fetch('section_name')) }:"
tail = render_list(children, depth + 1) if children.any?
[head, tail].compact.join("\n")
else
label = if node['tag']
node.fetch('tag').fetch('name')
else
"#{ TagGroupRepr::PREFIX }#{ node.fetch('tag_group').fetch('name') }"
end
raise UnsupportedStructure, '名称のない Placement は Markdown にできません.' if label.blank?
head = "#{ indent }- #{ clean_text(label) }"
children = node.fetch('children')
tail = render_list(children, depth + 1) if children.any?
[head, tail].compact.join("\n")
end
end.join("\n")
end
def clean_text value
value.to_s.gsub(/\r\n?/, "\n").lines.map(&:rstrip).join("\n").strip
end
end
+362
ファイルの表示
@@ -0,0 +1,362 @@
class TagGroupOperation
MAX_PLACEMENT_DEPTH = 5
def self.create!(locale, name:, description:, members:, placements:, actor:,
require_compatibility: false)
ApplicationRecord.transaction do
group = TagGroup.new(version_no: 1, created_by_user: actor,
updated_by_user: actor)
attributes = prepare!(locale:, group:, name:, description:, members:, placements:)
group.assign_attributes(name: attributes.fetch(:name),
description: attributes.fetch(:description))
group.save!
persist_structure!(group:, attributes:, actor:)
validate_compatibility!(locale, group) if require_compatibility
persist_version!(group:, attributes:, actor:, event_type: 'create')
group
end
end
def self.update!(locale, group:, name:, description:, members:, placements:, actor:,
require_compatibility: false)
ApplicationRecord.transaction do
group.lock!
attributes = prepare!(locale:, group:, name:, description:, members:, placements:)
if unchanged?(group, attributes)
validate_compatibility!(locale, group) if require_compatibility
next group
end
group.update!(name: attributes.fetch(:name),
description: attributes.fetch(:description),
version_no: group.version_no + 1,
updated_by_user: actor)
persist_structure!(group:, attributes:, actor:)
validate_compatibility!(locale, group) if require_compatibility
persist_version!(group:, attributes:, actor:, event_type: 'update')
group
end
end
def self.validate_compatibility! locale, group
TagGroupMarkdownRenderer.render(TagGroupRepr.one(locale, group))
rescue TagGroupMarkdownRenderer::UnsupportedStructure => error
invalid!(group, :base, error.message)
end
private_class_method :validate_compatibility!
def self.prepare!(locale:, group:, name:, description:, members:, placements:)
record = group
normalized_name = name.to_s.strip
invalid!(record, :name, 'グループ名は必須です.') if normalized_name.blank?
if normalized_name.length > 255
invalid!(record, :name, 'グループ名は 255 文字以内で入力してください.')
end
invalid!(record, :description, '説明文は文字列で入力してください.') unless
description.nil? || description.is_a?(String)
record.name = normalized_name
record.description = description
unless record.valid?
raise ActiveRecord::RecordInvalid, record
end
member_rows = normalize_members(record, members)
placement_tree = normalize_placements(record, placements)
validate_references!(locale, record, member_rows, placement_tree)
validate_containment!(record, member_rows)
{ name: normalized_name, description:, members: member_rows,
placements: placement_tree }
end
private_class_method :prepare!
def self.unchanged?(group, attributes)
unless (group.name_in_database == attributes.fetch(:name) &&
group.description_in_database == attributes.fetch(:description))
return false
end
members = TagGroupTag.where(tag_group_id: group.id).pluck(:tag_id)
.map { { 'tag_id' => _1 } }
members.concat(TagGroupGroup.where(tag_group_id: group.id)
.pluck(:child_tag_group_id).map { { 'tag_group_id' => _1 } })
requested_members = attributes.fetch(:members).map do |member|
member.except('version_no')
end
return false unless members.to_set == requested_members.to_set
rows = TagGroupPlacement.where(tag_group_id: group.id).order(:position)
.group_by(&:parent_position)
build = lambda do |parent_position|
rows.fetch(parent_position, []).map do |row|
node = if row.tag_id
{ 'tag_id' => row.tag_id }
elsif row.child_tag_group_id
{ 'tag_group_id' => row.child_tag_group_id }
else
{ 'section_name' => row.section_name, 'description' => row.description }
end
node.merge('children' => build.call(row.position))
end
end
build.call(nil) == comparable_placements(attributes.fetch(:placements))
end
private_class_method :unchanged?
def self.comparable_placements(nodes)
nodes.map do |node|
result = node.dup
result['description'] = node['description'] if node.key?('section_name')
result['children'] = comparable_placements(node.fetch('children'))
result
end
end
private_class_method :comparable_placements
def self.normalize_members(record, members)
invalid!(record, :members, 'members は配列で指定してください.') unless members.is_a?(Array)
result = members.map do |member|
member = stringify_keys(member)
unless member.is_a?(Hash) && member.keys.one? &&
(member.key?('tag_id') || member.key?('tag_group_id'))
invalid!(record, :members, '各 member は tag_id または tag_group_id を一つ指定してください.')
end
key = member.keys.first
id = integer_id(record, member.fetch(key), :members)
{ key => id }
end
invalid!(record, :members, 'membership に重複があります.') unless result.uniq == result
result
end
private_class_method :normalize_members
def self.normalize_placements(record, placements)
invalid!(record, :placements, 'placements は配列で指定してください.') unless
placements.is_a?(Array)
normalize_placement_nodes(record, placements, 1)
end
private_class_method :normalize_placements
def self.normalize_placement_nodes(record, nodes, depth)
if depth > MAX_PLACEMENT_DEPTH && nodes.any?
invalid!(record, :placements, '配置の深さは 5 階層までです.')
end
nodes.map do |raw_node|
node = stringify_keys(raw_node)
unless node.is_a?(Hash) && node.key?('children')
invalid!(record, :placements, '各 Placement に children 配列が必要です.')
end
children = node.delete('children')
unless children.is_a?(Array)
invalid!(record, :placements, 'children は配列で指定してください.')
end
kinds = ['tag_id', 'tag_group_id', 'section_name'].select { node.key?(_1) }
invalid!(record, :placements, 'Placement の種類を一つ指定してください.') unless
kinds.one?
kind = kinds.first
allowed = [kind, 'description']
invalid!(record, :placements, '不明な Placement 項目があります.') unless
(node.keys - allowed).empty?
case kind
when 'tag_id', 'tag_group_id'
node[kind] = integer_id(record, node.fetch(kind), :placements)
if node['description'].present?
invalid!(record, :placements, 'Tag / Tag Group Placement に説明文は指定できません.')
end
node.delete('description')
when 'section_name'
section_name = node.fetch(kind)
unless section_name.is_a?(String) && section_name.strip.present? &&
section_name.length <= 255
invalid!(record, :placements, 'Section 名は 1〜255 文字で入力してください.')
end
node[kind] = section_name.strip
unless node['description'].nil? || node['description'].is_a?(String)
invalid!(record, :placements, 'Section の説明文は文字列で入力してください.')
end
end
node['children'] = normalize_placement_nodes(record, children, depth + 1)
node
end
end
private_class_method :normalize_placement_nodes
def self.validate_references!(locale, record, members, placements)
membership = members.to_set
placed = placement_references(placements).uniq
missing = placed - membership.to_a
invalid!(record, :placements, 'Placement の対象は membership に含めてください.') if
missing.any?
tag_ids = members.filter_map { _1['tag_id'] }
group_ids = members.filter_map { _1['tag_group_id'] }
tags = Tag.where(id: tag_ids).index_by(&:id)
groups = TagGroup.where(id: group_ids).index_by(&:id)
invalid!(record, :members, '存在しない Tag が含まれています.') if
tags.size != tag_ids.uniq.size
invalid!(record, :members, '存在しない Tag Group が含まれています.') if
groups.size != group_ids.uniq.size
tag_versions = TagVersion.where(tag_id: tag_ids).pluck(:tag_id, :version_no).to_set
group_versions = TagGroupVersion.where(tag_group_id: group_ids)
.pluck(:tag_group_id, :version_no).to_set
tag_names = TagName.where(
tag_id: tag_ids, language_code: locale.language_code, primary_flg: true)
.pluck(:tag_id).uniq
members.each do |member|
if member.key?('tag_id')
id = member.fetch('tag_id')
invalid!(record, :members, :invalid) unless
tag_names.include?(id)
version_no = tags.fetch(id).version_no
unless tag_versions.include?([id, version_no])
invalid!(record, :members, "Tag ##{ id } の Version がありません.")
end
member['version_no'] = version_no
else
id = member.fetch('tag_group_id')
version_no = groups.fetch(id).version_no
unless group_versions.include?([id, version_no])
invalid!(record, :members, "Tag Group ##{ id } の Version がありません.")
end
member['version_no'] = version_no
end
end
end
private_class_method :validate_references!
def self.validate_containment!(record, members)
target_id = record.id
return unless target_id
children = members.filter_map { _1['tag_group_id'] }
edges = TagGroupGroup.pluck(:tag_group_id, :child_tag_group_id)
.group_by(&:first).transform_values { _1.map(&:last) }
edges[target_id] = children
visiting = Set.new
visited = Set.new
visit = lambda do |group_id|
invalid!(record, :members, 'Tag Group の循環包含はできません.') if
group_id == target_id && visiting.include?(group_id)
return if visited.include?(group_id)
visiting.add(group_id)
edges.fetch(group_id, []).each do |child_id|
invalid!(record, :members, 'Tag Group の循環包含はできません.') if
child_id == target_id
visit.call(child_id)
end
visiting.delete(group_id)
visited.add(group_id)
end
children.each { visit.call(_1) }
end
private_class_method :validate_containment!
def self.persist_structure!(group:, attributes:, actor:)
tag_members = TagGroupTag.where(tag_group_id: group.id).index_by(&:tag_id)
group_members = TagGroupGroup.where(tag_group_id: group.id).index_by(&:child_tag_group_id)
attributes.fetch(:members).each do |member|
if member.key?('tag_id')
next if tag_members.delete(member.fetch('tag_id'))
TagGroupTag.create!(tag_group_id: group.id, tag_id: member.fetch('tag_id'),
created_by_user: actor)
else
next if group_members.delete(member.fetch('tag_group_id'))
TagGroupGroup.create!(tag_group_id: group.id,
child_tag_group_id: member.fetch('tag_group_id'),
created_by_user: actor)
end
end
existing = TagGroupPlacement.where(tag_group_id: group.id).index_by(&:position)
persist_placements!(group:, nodes: attributes.fetch(:placements), actor:, existing:)
existing.values.sort_by(&:position).reverse_each(&:delete)
tag_members.each_value(&:delete)
group_members.each_value(&:delete)
end
private_class_method :persist_structure!
def self.persist_placements!(group:, nodes:, actor:, existing:, parent_position: nil,
next_position: [0])
nodes.each do |node|
next_position[0] += 1
position = next_position[0]
row = existing.delete(position) || TagGroupPlacement.new(
tag_group_id: group.id, position:, created_by_user: actor)
row.assign_attributes(
parent_position:,
section_name: node['section_name'], description: node['description'],
tag_id: node['tag_id'], child_tag_group_id: node['tag_group_id'])
if row.new_record? || row.changed?
row.updated_by_user = actor
row.save!
end
persist_placements!(group:, nodes: node.fetch('children'), actor:, existing:,
parent_position: position, next_position:)
end
end
private_class_method :persist_placements!
def self.persist_version!(group:, attributes:, actor:, event_type:)
TagGroupVersion.create!(tag_group: group, version_no: group.version_no,
event_type:, name: group.name,
description: group.description,
members: attributes.fetch(:members),
placements: attributes.fetch(:placements),
created_by_user: actor)
end
private_class_method :persist_version!
def self.placement_references(nodes)
nodes.flat_map do |node|
own = if node.key?('tag_id')
[{ 'tag_id' => node.fetch('tag_id') }]
elsif node.key?('tag_group_id')
[{ 'tag_group_id' => node.fetch('tag_group_id') }]
else
[]
end
own + placement_references(node.fetch('children'))
end
end
private_class_method :placement_references
def self.stringify_keys(value)
case value
when ActionController::Parameters
value.to_unsafe_h.deep_stringify_keys
when Hash
value.deep_stringify_keys
else
value
end
end
private_class_method :stringify_keys
def self.integer_id(record, value, attribute)
integer = if value.is_a?(Integer)
value
elsif value.is_a?(String) && value.match?(/\A[0-9]+\z/)
value.to_i
end
invalid!(record, attribute, 'ID は正の整数で指定してください.') unless integer
invalid!(record, attribute, 'ID は正の整数で指定してください.') unless integer.positive?
integer
end
private_class_method :integer_id
def self.invalid!(record, attribute, message)
record.errors.add(attribute, message)
raise ActiveRecord::RecordInvalid, record
end
private_class_method :invalid!
end
+122
ファイルの表示
@@ -0,0 +1,122 @@
class TagNameCompletion
class << self
def for_locale! locale
completed = Tag.transaction do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
certified_names =
if locale.certified_tag_names_path.nil?
{ }
else
CertifiedTagNameTranslations.load!(locale.certified_tag_names_path, locale:)
end
Tag.where(id: certified_names.keys).lock.find_each(order: :asc) do |tag|
for_tag!(tag, locales: [locale],
certified_names: { locale.language_code => certified_names.fetch(tag.id) },
created_by_user: nil)
end
named_tags =
TagName.where(language_code: locale.language_code, primary_flg: true)
.select(:tag_id)
Tag.where.not(id: named_tags).lock.find_each(order: :asc) do |tag|
for_tag!(tag, locales: [locale], created_by_user: nil)
end
true
end
unless completed
raise ActiveRecord::RecordNotSaved, 'Locale tag completion was rolled back'
end
end
def for_tag! tag, locales: nil, certified_names: { }, created_by_user: nil
locales ||=
Locale.where(deprecated_at: nil)
.order(:language_code, :created_at, :script_code).lock
Tag.transaction do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
# Creation includes every language in its first snapshot.
creating = tag.new_record?
tag.save! if creating
completed = tag.with_lock do
primary_names = tag.tag_names.where(primary_flg: true).lock.index_by(&:language_code)
certified_names.each do |language, name|
primary = primary_names[language]
next unless primary
next if primary.name == name && !primary.auto_generated
raise CertifiedTagNameTranslations::Error,
"Tag##{ tag.id } already has a primary #{ language } name; " +
"cannot register certified name #{ name.inspect }"
end
missing_locales = locales.to_a.uniq(&:language_code).reject { |locale|
primary_names.key?(locale.language_code)
}
next true if missing_locales.empty? && !creating
TagVersioning.ensure_snapshot!(tag, created_by_user:) unless creating
missing_locales.each do |locale|
primary_names[locale.language_code] =
if certified_names.key?(locale.language_code)
create_certified_name!(locale, tag, certified_names.fetch(locale.language_code))
else
source_language = locale.language_code == 'ja' ? 'ain' : 'ja'
name = TagName.generate_name(locale, tag, primary_names[source_language])
create_name!(locale, tag, name)
end
end
tag.association(:tag_names).reset
event_type = creating ? :create : :update
version = TagVersioning.record!(tag, event_type:, created_by_user:)
unless version&.persisted?
raise ActiveRecord::RecordNotSaved, 'Tag version was not persisted'
end
true
end
unless completed
raise ActiveRecord::RecordNotSaved, 'Tag completion was rolled back'
end
end
end
private
def create_certified_name! locale, tag, name
persist_name!(locale, tag, name, auto_generated: false)
rescue ActiveRecord::RecordInvalid => error
raise unless error.record.is_a?(TagName)
raise CertifiedTagNameTranslations::Error,
"Tag##{ tag.id } certified name #{ name.inspect }: " +
error.record.errors.full_messages.join(', ')
rescue ActiveRecord::RecordNotUnique => error
raise CertifiedTagNameTranslations::Error,
"Tag##{ tag.id } certified name #{ name.inspect } conflicts: #{ error.message }"
end
def create_name! locale, tag, name
persist_name!(locale, tag, name, auto_generated: true)
rescue ActiveRecord::RecordNotUnique, ActiveRecord::RecordInvalid => error
fallback = "Tag_##{ tag.id }"
invalid_name =
if error.is_a?(ActiveRecord::RecordNotUnique)
TagName.where(language_code: locale.language_code, name:).lock.exists?
else
error.record.errors[:name].present? &&
error.record.errors.attribute_names.all? { _1 == :name }
end
raise if name == fallback || !(invalid_name)
persist_name!(locale, tag, fallback, auto_generated: true)
end
def persist_name! locale, tag, name, auto_generated:
tag_name = TagName.transaction(requires_new: true) do
TagName.create!(tag:, name:, language_code: locale.language_code,
script_code: locale.script_code, primary_flg: true, auto_generated:)
end
unless tag_name&.persisted?
raise ActiveRecord::RecordNotSaved, 'Generated tag name was not persisted'
end
tag_name
end
end
end
+56
ファイルの表示
@@ -0,0 +1,56 @@
module TagNameTranscription
class Unconvertible < StandardError
;
end
def self.ainu name
previous_script = nil
parts = []
NipponeseSegments.split(name).each do |script, text|
output = transcribe_segment(script, text)
if (parts.last && output.first && previous_script && script != previous_script &&
NipponeseSegments.nipponese?(previous_script) && NipponeseSegments.nipponese?(script))
parts << '_'
end
parts.concat(output)
previous_script = script
end
replace_unconvertible(parts)
end
def self.transcribe_segment script, text
return [text] if script == :literal
return [nil] if script == :unconvertible
return transcribe_reading(text) unless script == :kanzi
text.each_char.flat_map do |character|
transcribe_reading(JisReadings.read(character))
rescue Unconvertible
[nil]
end
end
def self.transcribe_reading reading
Phonology.adapt(reading).map do |sound|
sound && Katakana.render([sound])
rescue Unconvertible
nil
end
end
def self.replace_unconvertible parts
grouped = parts.chunk { _1.nil? }.flat_map { |unconvertible, group|
unconvertible ? [nil] : group
}
first = grouped.index { _1&.match?(/[^_]/) }
last = grouped.rindex { _1&.match?(/[^_]/) }
grouped.each_with_index.map { |part, index|
part || (first && first < index && index < last ? '_' : '')
}.join
end
private_class_method :transcribe_segment, :transcribe_reading, :replace_unconvertible
end
+13
ファイルの表示
@@ -0,0 +1,13 @@
module TagNameTranscription
module JisReadings
SOURCE = Rails.root.join('config/tag_name_readings/jis_level_one.tsv')
READINGS = File.foreach(SOURCE, chomp: true).reject { _1.start_with?('#') }.to_h { |line|
character, reading, = line.split("\t")
[character, reading]
}.freeze
def self.read text
text.each_char.map { READINGS.fetch(_1) { raise Unconvertible } }.join
end
end
end
+26
ファイルの表示
@@ -0,0 +1,26 @@
module TagNameTranscription
module Katakana
ROWS = {
'' => 'アイウエオ',
'k' => 'カキクケコ',
's' => 'サシスセソ',
'c' => ['チャ', 'チ', 'チュ', 'チェ', 'チョ'],
't' => ['タ', 'ティ', 'トゥ', 'テ', 'ト'],
'n' => 'ナニヌネノ',
'h' => 'ハヒフヘホ',
'p' => 'パピプペポ',
'm' => 'マミムメモ',
'r' => 'ラリルレロ' }.freeze
SYLLABLES = ROWS.flat_map { |onset, letters|
kana = letters.is_a?(String) ? letters.chars : letters
['a', 'i', 'u', 'e', 'o'].zip(kana).map { |vowel, letter| [onset + vowel, letter] }
}.to_h.merge(
'ya' => 'ヤ', 'yu' => 'ユ', 'yo' => 'ヨ', 'wa' => 'ワ',
'k' => 'ㇰ', 's' => 'ㇱ', 't' => 'ッ', 'n' => 'ン', 'm' => 'ㇺ', 'p' => 'ㇷ゚',
'r:a' => 'ㇻ', 'r:i' => 'ㇼ', 'r:u' => 'ㇽ', 'r:e' => 'ㇾ', 'r:o' => 'ㇿ').freeze
def self.render sounds
sounds.map { SYLLABLES.fetch(_1) { raise Unconvertible } }.join
end
end
end
+30
ファイルの表示
@@ -0,0 +1,30 @@
module TagNameTranscription
module NipponeseSegments
NIPPONESE_SCRIPTS = [:kanzi, :xiragana, :katakana].freeze
def self.nipponese?(script) = NIPPONESE_SCRIPTS.include?(script)
def self.split name
segments = []
name.to_s.scan(/\X/).each do |character|
script = classify(character, segments.last&.first)
if segments.last&.first == script
segments.last[1] << character
else
segments << [script, character.dup]
end
end
segments
end
def self.classify character, previous_script
return previous_script if character == 'ー' && nipponese?(previous_script)
return :kanzi if character.match?(/\A\p{Han}\z/)
return :xiragana if character.match?(/\A\p{Hiragana}\p{M}*\z/)
return :katakana if character.match?(/\A\p{Katakana}\p{M}*\z/)
return :literal if character.match?(/\A(?:\p{Latin}\p{M}*|[!-~])\z/)
:unconvertible
end
end
end
+59
ファイルの表示
@@ -0,0 +1,59 @@
module TagNameTranscription
module Phonology
ROWS = {
'アイウエオ' => ['a', 'i', 'u', 'e', 'o'],
'カキクケコ' => ['ka', 'ki', 'ku', 'ke', 'ko'],
'ガギグゲゴ' => ['ka', 'ki', 'ku', 'ke', 'ko'],
'サシスセソ' => ['sa', 'si', 'su', 'se', 'so'],
'ザジズゼゾ' => ['ca', 'ci', 'cu', 'ce', 'co'],
'タチツテト' => ['ta', 'ci', 'cu', 'te', 'to'],
'ダヂヅデド' => ['ta', 'ci', 'cu', 'te', 'to'],
'ナニヌネノ' => ['na', 'ni', 'nu', 'ne', 'no'],
'ハヒフヘホ' => ['ha', 'hi', 'hu', 'he', 'ho'],
'バビブベボ' => ['pa', 'pi', 'pu', 'pe', 'po'],
'パピプペポ' => ['pa', 'pi', 'pu', 'pe', 'po'],
'マミムメモ' => ['ma', 'mi', 'mu', 'me', 'mo'],
'ヤユヨ' => ['ya', 'yu', 'yo'],
'ラリルレロ' => ['ra', 'ri', 'ru', 're', 'ro'],
'ワ' => ['wa'],
'ン' => ['n'],
'ッ' => ['t'],
'ㇰ' => ['k'],
'ㇱㇲ' => ['s', 's'],
'ㇳ' => ['t'],
'ㇴ' => ['n'],
'ㇺ' => ['m'],
'ㇻㇼㇽㇾㇿ' => ['r:a', 'r:i', 'r:u', 'r:e', 'r:o'] }.freeze
SOUNDS = ROWS.flat_map { |characters, sounds| characters.chars.zip(sounds) }.to_h.freeze
DIGRAPHS = {
'シャ' => 'sa', 'シュ' => 'su', 'シェ' => 'se', 'ショ' => 'so',
'チャ' => 'ca', 'チュ' => 'cu', 'チェ' => 'ce', 'チョ' => 'co',
'ジャ' => 'ca', 'ジュ' => 'cu', 'ジェ' => 'ce', 'ジョ' => 'co',
'ヂャ' => 'ca', 'ヂュ' => 'cu', 'ヂェ' => 'ce', 'ヂョ' => 'co' }.freeze
def self.adapt reading
kana = reading.unicode_normalize(:nfc).tr('ぁ-ゖ', 'ァ-ヶ').delete('ー')
characters = kana.scan(/\X/)
sounds = []
index = 0
while index < characters.length
pair = characters[index, 2].join
if DIGRAPHS.key?(pair)
sounds << DIGRAPHS.fetch(pair)
index += 2
next
end
character = characters[index].tr('ャュョ', 'ヤユヨ')
sounds << (character == 'ㇷ゚' ? 'p' : SOUNDS[character])
index += 1
end
sounds
end
end
end
+22 -6
ファイルの表示
@@ -1,6 +1,9 @@
class TagVersionRecorder < VersionRecorder
def self.record! tag:, event_type:, created_by_user:, **_options
new(tag:, event_type:, created_by_user:).record!
Tag.transaction do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
new(tag:, event_type:, created_by_user:).record!
end
end
def initialize tag:, event_type:, created_by_user:
@@ -15,26 +18,39 @@ class TagVersionRecorder < VersionRecorder
def snapshot_attributes
names = snapshot_names
compatibility_names = names.fetch('ja')
compatibility_names = names['ja'] || names['ain'] || names.values.first
unless compatibility_names
@record.errors.add(:tag_names, :invalid)
raise ActiveRecord::RecordInvalid, @record
end
parent_ids =
TagImplication.where(tag_id: @record.id).order(:parent_tag_id)
.lock.pluck(:parent_tag_id)
{ name: compatibility_names.fetch('name'),
category: @record.category,
deprecated_at: @record.deprecated_at,
aliases: compatibility_names.fetch('aliases').map { _1.fetch('name') }.join(' '),
names:,
parent_tag_ids: @record.snapshot_parent_tag_ids.join(' ') }
parent_tag_ids: parent_ids.join(' ') }
end
def snapshot_names
@record.tag_names.order(:language_code, :name).group_by(&:language_code)
@record.tag_names.order(:language_code, :name).lock.group_by(&:language_code)
.transform_values do |tag_names|
primary = tag_names.find(&:primary?)
unless primary
@record.errors.add(:tag_names, :invalid,
language_code: tag_names.first.language_code)
raise ActiveRecord::RecordInvalid, @record
end
aliases = tag_names.reject(&:primary?).sort_by(&:name).map { |tag_name|
{ 'name' => tag_name.name, 'script_code' => tag_name.script_code }
{ 'name' => tag_name.name,
'script_code' => Script.codes.fetch(tag_name.script_code, tag_name.script_code) }
}
{ 'name' => primary.name,
'script_code' => primary.script_code,
'script_code' => Script.codes.fetch(primary.script_code, primary.script_code),
'auto_generated' => primary.auto_generated,
'aliases' => aliases }
end
+13 -4
ファイルの表示
@@ -4,14 +4,23 @@ class TagVersioning
end
def self.ensure_snapshot! tag, created_by_user:, language_code: 'ja'
return if tag.tag_versions.exists?
Tag.transaction do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
locked_tag = Tag.unscoped.lock.find(tag.id)
next if locked_tag.tag_versions.lock.exists?
TagVersionRecorder.record!(tag:, event_type: :create, created_by_user:, language_code:)
TagVersionRecorder.record!(
tag: locked_tag, event_type: :create, created_by_user:, language_code:)
end
end
def self.record_tag_snapshot! tag, created_by_user:, language_code: 'ja'
event_type = tag.tag_versions.exists? ? :update : :create
record!(tag, event_type:, created_by_user:, language_code:)
Tag.transaction do
SharedResourceIdAllocator.lock_space!(Tag, 'tags')
locked_tag = Tag.unscoped.lock.find(tag.id)
event_type = locked_tag.tag_versions.lock.exists? ? :update : :create
record!(locked_tag, event_type:, created_by_user:, language_code:)
end
end
def self.record_tag_snapshots! tags, created_by_user:, language_code: 'ja'
+1 -1
ファイルの表示
@@ -35,7 +35,7 @@ class VersionRecorder
private
def latest_version = versions.order(version_no: :desc).first
def latest_version = versions.order(version_no: :desc).lock.first
def versions = @record.public_send(version_association)
-160
ファイルの表示
@@ -1,160 +0,0 @@
require 'digest'
module Wiki
class Commit
class Conflict < StandardError
;
end
def self.create_content! tag_name:, body:, created_by_user:, message: nil
if tag_name.tag && tag_name.language_code == 'ja'
return TagDescriptionOperation.create!(
tag: tag_name.tag, body:, created_by_user:, message:)
end
create_legacy_content!(tag_name:, body:, created_by_user:, message:)
end
def self.create_legacy_content! tag_name:, body:, created_by_user:, message: nil
normalised = normalise_body(body)
page = WikiPage.new(tag_name:,
body: normalised,
created_user: created_by_user,
updated_user: created_by_user)
if normalised.blank?
page.errors.add(:body, :blank)
raise ActiveRecord::RecordInvalid, page
end
ActiveRecord::Base.transaction do
page.save!
new(page:, created_user: created_by_user).content!(
body: normalised,
message:,
base_revision_id: nil)
page
end
end
def self.content! page:, body:, created_user:, message: nil, base_revision_id: nil
description = TagDescription.find_by(id: page.id)
if description
return TagDescriptionOperation.update!(
tag: description.tag, body:, updated_by_user: created_user,
message:, base_revision_id:)
end
legacy_content!(page:, body:, created_user:, message:, base_revision_id:)
end
def self.legacy_content! page:, body:, created_user:, message: nil,
base_revision_id: nil
new(page:, created_user:).content!(body:, message:, base_revision_id:)
end
def self.redirect! page:, redirect_page:, created_user:, message: nil, base_revision_id: nil
new(page:, created_user:).redirect!(redirect_page:, message:, base_revision_id:)
end
def initialize page:, created_user:
@page = page
@created_user = created_user
end
def content! body:, message:, base_revision_id:
normalised = self.class.normalise_body(body)
if normalised.blank?
@page.errors.add(:body, :blank)
raise ActiveRecord::RecordInvalid, @page
end
lines = split_lines(normalised)
line_shas = lines.map { |line| Digest::SHA256.hexdigest(line) }
tree_sha = Digest::SHA256.hexdigest(line_shas.join(','))
ActiveRecord::Base.transaction do
@page.lock!
if base_revision_id.present?
current_id = @page.wiki_revisions.maximum(:id)
if current_id && current_id != base_revision_id.to_i
raise Conflict,
"競合が発生してゐます" +
"(現在の Id.:#{ current_id },ベース Id.:#{ base_revision_id })."
end
end
line_id_by_sha = upsert_lines!(lines, line_shas)
line_ids = line_shas.map { |sha| line_id_by_sha.fetch(sha) }
@page.update!(body: normalised)
WikiVersionRecorder.record!(
page: @page,
event_type: @page.wiki_versions.exists? ? :update : :create,
reason: message,
created_by_user: @created_user)
rev = WikiRevision.create!(
wiki_page: @page,
base_revision_id:,
created_user: @created_user,
kind: :content,
redirect_page_id: nil,
message:,
lines_count: lines.length,
tree_sha256: tree_sha)
rows = line_ids.each_with_index.map do |line_id, pos|
{ wiki_revision_id: rev.id, wiki_line_id: line_id, position: pos }
end
WikiRevisionLine.insert_all!(rows) if rows.any?
rev
end
end
def redirect!(redirect_page:, message:, base_revision_id:) = raise '廃止しました.'
def self.normalise_body body
s = body.to_s
s.gsub!(/\r\n?/, "\n")
s.encode('UTF-8', invalid: :replace, undef: :replace, replace: '🖕')
s.gsub(/\n+$/, '')
end
private
def split_lines(body) = body.split("\n")
def upsert_lines! lines, line_shas
now = Time.current
id_by_sha = WikiLine.where(sha256: line_shas).pluck(:sha256, :id).to_h
missing_by_sha = { }
line_shas.each_with_index do |sha, i|
next if id_by_sha.key?(sha)
next if missing_by_sha.key?(sha)
missing_by_sha[sha] = {
sha256: sha,
body: lines[i],
created_at: now,
updated_at: now }
end
if missing_by_sha.any?
WikiLine.upsert_all(missing_by_sha.values)
id_by_sha = WikiLine.where(sha256: line_shas).pluck(:sha256, :id).to_h
end
id_by_sha
end
end
end
-33
ファイルの表示
@@ -1,33 +0,0 @@
module Wiki
class CompatibilityWriter
def self.create! tag_name:, body:, created_by_user:, message: nil
Commit.create_legacy_content!(
tag_name:, body:, created_by_user:, message:)
end
def self.update! page:, body:, updated_by_user:, message: nil,
base_revision_id: nil
normalised = Commit.normalise_body(body)
if normalised.blank?
return Commit.legacy_content!(
page:, body: normalised, created_user: updated_by_user, message:,
base_revision_id:)
end
if base_revision_id.present?
current_id = page.wiki_revisions.maximum(:id)
if current_id && current_id != base_revision_id.to_i
raise Commit::Conflict,
"競合が発生してゐます" +
"(現在の Id.:#{ current_id },ベース Id.:#{ base_revision_id })."
end
end
return if Commit.normalise_body(page.body) == normalised
Commit.legacy_content!(
page:, body: normalised, created_user: updated_by_user, message:,
base_revision_id:)
end
end
end
-21
ファイルの表示
@@ -1,21 +0,0 @@
class WikiVersionRecorder < VersionRecorder
def self.record! page:, event_type:, reason: nil, created_by_user:
new(page:, event_type:, reason:, created_by_user:).record!
end
def initialize page:, event_type:, reason: nil, created_by_user:
@reason = reason
super(record: page, event_type:, created_by_user:)
end
private
def version_class = WikiVersion
def version_association = :wiki_versions
def record_key = :wiki_page
def snapshot_attributes = {
title: @record.title,
body: @record.body,
reason: @reason }
end
+9
ファイルの表示
@@ -31,5 +31,14 @@ module Backend
# Middleware like session, flash, cookies can be added back manually.
# Skip views, helpers and assets when generating a new resource.
config.api_only = true
config.middleware.use ActionDispatch::Cookies
config.session_store :cookie_store,
key: Rails.env.production? ? '__Host-btrc-session' : 'btrc-session',
path: '/',
secure: Rails.env.production?,
httponly: true,
same_site: :lax,
expire_after: 20.years
config.middleware.use ActionDispatch::Session::CookieStore, config.session_options
end
end
+2 -2
ファイルの表示
@@ -4,5 +4,5 @@
# Use this to limit dissemination of sensitive information.
# See the ActiveSupport::ParameterFilter documentation for supported notations and behaviors.
Rails.application.config.filter_parameters += [
:passw, :email, :secret, :token, :_key, :crypt, :salt, :certificate, :otp, :ssn, :cvv, :cvc
]
:passw, :email, :secret, :token, :_key, :crypt, :salt, :certificate,
:otp, :ssn, :cvv, :cvc, :code, :inheritance_code]
+21
ファイルの表示
@@ -1,4 +1,19 @@
Rails.application.routes.draw do
get 'counts/definitions/:resource', to: 'counts_tables#definition'
get 'counts/query', to: 'counts_tables#index'
post 'counts/export/prepare', to: 'counts_exports#prepare'
get 'counts/export/files/:token', to: 'counts_exports#download'
get 'counts/export', to: 'counts_exports#index'
post 'counts/export', to: 'counts_exports#index'
get 'counts/state', to: 'counts_states#index'
get 'counts/posts', to: 'counts_posts#index'
get 'counts/tags', to: 'counts_tags#index'
resources :locales, only: [:index]
resources :sitemap_entries, only: [:index, :show], param: :resource
resources :tag_groups, only: [:index, :show, :create, :update]
resources :nico_tags, path: 'tags/nico', only: [:index, :show, :update]
scope 'tags/:parent_id/children', controller: :tag_children do
@@ -41,6 +56,7 @@ Rails.application.routes.draw do
collection do
get :search, action: :index
get :changes
get :autolink_candidates
scope :title do
get ':title/exists', action: :exists_by_title
@@ -51,12 +67,14 @@ Rails.application.routes.draw do
member do
get :exists
get :diff
get :tag, action: :tag_reference
end
end
resources :posts, only: [:index, :show, :create, :update] do
collection do
get :metadata
get :tag_filters
post :bulk
get :random
get :versions, to: 'post_versions#index'
@@ -95,6 +113,9 @@ Rails.application.routes.draw do
collection do
post :verify
get :me
get :session, action: :show_session
get :csrf
delete :session, action: :destroy_session
post 'code/renew', action: :renew
end
end
ファイル差分が大きすぎるため省略します 差分を読み込み
+149
ファイルの表示
@@ -0,0 +1,149 @@
class CreateTagGroups < ActiveRecord::Migration[8.0]
def change
# 翻訳系は上流の Translation リソースが未完のため保留
create_table :tag_groups do |t|
t.integer :version_no, null: false
t.string :name, limit: 255, null: false
t.text :description
t.references :created_by_user, foreign_key: { to_table: :users }
t.references :updated_by_user, foreign_key: { to_table: :users }
t.timestamps
t.check_constraint 'version_no > 0', name: 'chk_tag_groups_version_no_positive'
t.index :name, unique: true
end
create_table :tag_group_tags, primary_key: [:tag_group_id, :tag_id] do |t|
t.references :tag_group, null: false, foreign_key: true, index: false
t.references :tag, null: false, foreign_key: true
t.references :created_by_user, foreign_key: { to_table: :users }
t.datetime :created_at, null: false
end
create_table :tag_group_groups, primary_key: [:tag_group_id, :child_tag_group_id] do |t|
t.references :tag_group, null: false, foreign_key: true, index: false
t.references :child_tag_group,
null: false, foreign_key: { to_table: :tag_groups }
t.references :created_by_user, foreign_key: { to_table: :users }
t.datetime :created_at, null: false
t.check_constraint 'tag_group_id <> child_tag_group_id',
name: 'chk_tag_group_groups_tag_group_id_not_child_tag_group_id'
end
create_table :tag_group_placements, primary_key: [:tag_group_id, :position] do |t|
t.references :tag_group, null: false, foreign_key: true, index: false
t.integer :position, null: false
t.integer :parent_position
t.string :section_name, limit: 255
t.text :description
t.references :tag
t.references :child_tag_group
t.references :created_by_user, foreign_key: { to_table: :users }
t.references :updated_by_user, foreign_key: { to_table: :users }
t.timestamps
t.foreign_key :tag_group_placements,
column: [:tag_group_id, :parent_position],
primary_key: [:tag_group_id, :position]
t.foreign_key :tag_group_tags,
column: [:tag_group_id, :tag_id],
primary_key: [:tag_group_id, :tag_id]
t.foreign_key :tag_group_groups,
column: [:tag_group_id, :child_tag_group_id],
primary_key: [:tag_group_id, :child_tag_group_id]
t.check_constraint 'position > parent_position',
name: 'chk_tag_group_placements_position_gt_parent_position'
t.check_constraint ('((section_name IS NOT NULL AND tag_id IS NULL AND child_tag_group_id IS NULL) OR ' +
'(section_name IS NULL AND tag_id IS NOT NULL AND child_tag_group_id IS NULL) OR ' +
'(section_name IS NULL AND tag_id IS NULL AND child_tag_group_id IS NOT NULL)) AND ' +
'(section_name IS NOT NULL OR description IS NULL)'),
name: 'chk_tag_group_placements_references_exclusive'
end
create_table :tag_group_versions do |t|
t.references :tag_group, null: false
t.integer :version_no, null: false
t.string :event_type, null: false
t.string :name, limit: 255, null: false
t.text :description
t.json :members, null: false
t.json :placements, null: false
t.references :created_by_user
t.datetime :created_at, null: false
t.check_constraint 'version_no > 0',
name: 'chk_tag_group_versions_version_no_positive'
t.index [:tag_group_id, :version_no], unique: true
add_members_constraint!(t)
add_placements_constraint!(t)
end
end
private
def add_members_constraint! t
schema = { type: 'array',
uniqueItems: true,
items: member_schema }
quoted_schema = connection.quote(JSON.generate(schema))
t.check_constraint "JSON_SCHEMA_VALID(#{ quoted_schema }, members)",
name: 'chk_tag_group_versions_members_schema'
end
def member_schema
{ oneOf: [tag_member_schema, group_member_schema] }
end
def tag_member_schema
{ type: 'object',
required: ['tag_id', 'version_no'],
properties: { tag_id: { type: 'integer', minimum: 1 },
version_no: { type: 'integer', minimum: 1 } },
additionalProperties: false }
end
def group_member_schema
{ type: 'object',
required: ['tag_group_id', 'version_no'],
properties: { tag_group_id: { type: 'integer', minimum: 1 },
version_no: { type: 'integer', minimum: 1 } },
additionalProperties: false }
end
def add_placements_constraint! t
schema = { type: 'array', items: placement_schema(5) }
quoted_schema = connection.quote(JSON.generate(schema))
t.check_constraint "JSON_SCHEMA_VALID(#{ quoted_schema }, placements)",
name: 'chk_tag_group_versions_placements_schema'
end
def placement_schema rest
{ type: 'object',
required: ['children'],
properties: { section_name: { type: 'string' },
description: { type: ['string', 'null'] },
tag_id: { type: 'integer', minimum: 1 },
tag_group_id: { type: 'integer', minimum: 1 },
children: if rest > 1
{ type: 'array', items: placement_schema(rest - 1) }
else
{ type: 'array', maxItems: 0 }
end },
oneOf: [{ required: ['section_name'] },
{ required: ['tag_id'] },
{ required: ['tag_group_id'] }],
dependencies: { description: ['section_name'] },
additionalProperties: false }
end
end
+518
ファイルの表示
@@ -0,0 +1,518 @@
class MigrateTagGroupsFromWikiPages < ActiveRecord::Migration[8.0]
PREFIX = 'タググループ:'
def up
require 'json'
require 'time'
require 'kramdown'
require 'kramdown-parser-gfm'
assert_empty_destination!
pages = connection.select_all(<<~SQL).to_a
SELECT
wp.id
, wp.created_user_id
, wp.updated_user_id
, wp.created_at
, wp.updated_at
, wp.discarded_at
, tn.name
FROM
wiki_pages wp
INNER JOIN
tag_names tn
ON
tn.id = wp.tag_name_id
WHERE
tn.language_code = 'ja'
AND tn.name LIKE #{ quote("#{ PREFIX }%") }
ORDER BY
wp.id
SQL
return if pages.empty?
pages.each do |page|
if page['discarded_at']
say "Wiki ##{ page.fetch('id') }: 破棄済みですが,Revision の移行を継続します."
end
end
tag_ids = load_tag_ids
page_ids = pages.map { |page| page.fetch('id').to_i }
revisions = connection.select_all(<<~SQL).to_a
SELECT
wr.id
, wr.wiki_page_id
, wr.kind
, wr.redirect_page_id
, wr.created_user_id
, wr.created_at
FROM
wiki_revisions wr
WHERE
wr.wiki_page_id IN (#{ page_ids.join(', ') })
ORDER BY
wr.wiki_page_id
, wr.id
SQL
revisions_by_page = revisions.group_by { |row| row.fetch('wiki_page_id').to_i }
pages = pages.select do |page|
page_id = page.fetch('id').to_i
if revisions_by_page.fetch(page_id, []).empty?
# tag_groups.version_no > 0 のため,Revision 0 件の Wiki は作成しない.
say "Wiki ##{ page_id }: Revision が 0 件のため移行対象から除外します."
false
else
true
end
end
return if pages.empty?
group_ids = pages.to_h { |page| [page.fetch('name'), page.fetch('id').to_i] }
revision_history = pages.to_h do |page|
page_id = page.fetch('id').to_i
history = revisions_by_page.fetch(page_id).each_with_index.map do |revision, index|
{ 'id' => revision.fetch('id').to_i,
'version_no' => index + 1,
'created_at' => revision.fetch('created_at') }
end
[page_id, history]
end
tag_history = connection.select_all(<<~SQL).to_a.group_by { |row| row.fetch('tag_id').to_i }
SELECT
tag_id
, version_no
, created_at
FROM
tag_versions
ORDER BY
tag_id
, created_at
, version_no
SQL
prepared = { }
pages.each do |page|
page_id = page.fetch('id').to_i
history = revisions_by_page.fetch(page_id)
prepared[page_id] = history.each_with_index.map do |revision, index|
context = "Wiki ##{ page_id } / Revision ##{ revision.fetch('id') }"
body = revision_body(revision.fetch('id').to_i)
if revision.fetch('kind').to_i == 1
destination = revision['redirect_page_id']
description = "旧 Wiki のリダイレクト先: Wiki ##{ destination }"
placements = []
members = []
say "#{ context }: リダイレクトを説明文として移行します."
else
begin
description, placements = parse_wiki(body, tag_ids, group_ids, context)
validate_tree!(placements, context)
members = build_members(placements, revision.fetch('created_at'),
tag_history, revision_history, context)
rescue StandardError => error
# 解析できない版も捨てず,当時の Markdown を説明文として保存する.
say "#{ context }: 構造化を保留し,本文を保存します (#{ error.message })."
description = body
placements = []
members = []
end
end
{ revision:, version_no: index + 1, description:, members:, placements:, body: }
end
end
# 自己包含は禁止されてゐるため,当該版は原文の保存を優先する.
prepared.each do |group_id, versions|
current = versions.last
next unless current.fetch(:members).any? { |member| member['tag_group_id'] == group_id }
say "Tag Group ##{ group_id }: 直接の自己参照を検出したため,現行構造化を保留します."
current[:description] = current.fetch(:body)
current[:members] = []
current[:placements] = []
end
begin
validate_current_groups!(prepared)
rescue StandardError => error
say "Tag Group の包含関係に問題がありますが,移行を継続します (#{ error.message })."
end
# 移行先の書込みはすべて検証の後.DDL は含めない.
connection.transaction do
pages.each do |page|
page_id = page.fetch('id').to_i
current = prepared.fetch(page_id).last
insert_row('tag_groups', {
id: page_id,
# page.version_no ではなく,この Wiki に紐付く Revision の総数.
version_no: revisions_by_page.fetch(page_id).size,
name: page.fetch('name').delete_prefix(PREFIX),
description: current.fetch(:description),
created_by_user_id: page.fetch('created_user_id'),
updated_by_user_id: page.fetch('updated_user_id'),
created_at: page.fetch('created_at'),
updated_at: page.fetch('updated_at') })
end
pages.each do |page|
page_id = page.fetch('id').to_i
name = page.fetch('name').delete_prefix(PREFIX)
prepared.fetch(page_id).each do |item|
revision = item.fetch(:revision)
insert_row('tag_group_versions', {
id: revision.fetch('id'),
tag_group_id: page_id,
version_no: item.fetch(:version_no),
event_type: item.fetch(:version_no) == 1 ? 'create' : 'update',
name:,
description: item.fetch(:description),
members: JSON.generate(item.fetch(:members)),
placements: JSON.generate(item.fetch(:placements)),
created_by_user_id: revision.fetch('created_user_id'),
created_at: revision.fetch('created_at') })
end
end
pages.each do |page|
page_id = page.fetch('id').to_i
current = prepared.fetch(page_id).last
revision = current.fetch(:revision)
current.fetch(:members).each do |member|
if member.key?('tag_id')
insert_row('tag_group_tags', {
tag_group_id: page_id,
tag_id: member.fetch('tag_id'),
created_by_user_id: revision.fetch('created_user_id'),
created_at: revision.fetch('created_at') })
else
insert_row('tag_group_groups', {
tag_group_id: page_id,
child_tag_group_id: member.fetch('tag_group_id'),
created_by_user_id: revision.fetch('created_user_id'),
created_at: revision.fetch('created_at') })
end
end
end
pages.each do |page|
page_id = page.fetch('id').to_i
current = prepared.fetch(page_id).last
revision = current.fetch(:revision)
position = 0
append = -> nodes, parent_position do
nodes.each do |node|
position += 1
own_position = position
insert_row('tag_group_placements', {
tag_group_id: page_id,
position: own_position,
parent_position:,
section_name: node['section_name'],
description: node['description'],
tag_id: node['tag_id'],
child_tag_group_id: node['tag_group_id'],
created_by_user_id: revision.fetch('created_user_id'),
updated_by_user_id: revision.fetch('created_user_id'),
created_at: revision.fetch('created_at'),
updated_at: revision.fetch('created_at') })
append.(node.fetch('children'), own_position)
end
end
append.(current.fetch(:placements), nil)
end
end
end
def down
raise ActiveRecord::IrreversibleMigration, '戻せません.'
end
private
def assert_empty_destination!
['tag_groups', 'tag_group_versions', 'tag_group_tags',
'tag_group_groups', 'tag_group_placements'].each do |table|
count = connection.select_value("SELECT COUNT(*) FROM #{ table }").to_i
raise "#{ table } が空ではありません." if count.positive?
end
end
# 解決は tag_names.name -> tag_names.tag_id のみ.
# tags.tag_name_id は読まない.
def load_tag_ids
rows = connection.select_all(<<~SQL).to_a
SELECT
name
, tag_id
FROM
tag_names
WHERE
language_code = 'ja'
AND tag_id IS NOT NULL
SQL
rows.each_with_object({ }) do |row, by_name|
name = row.fetch('name')
id = row.fetch('tag_id').to_i
by_name[name] ||= []
by_name[name] << id unless by_name[name].include?(id)
end
end
def revision_body revision_id
connection.select_rows(<<~SQL).flatten.join("\n")
SELECT
wl.body
FROM
wiki_revision_lines wrl
INNER JOIN wiki_lines wl ON wl.id = wrl.wiki_line_id
WHERE
wrl.wiki_revision_id = #{ Integer(revision_id) }
ORDER BY
wrl.position
SQL
end
def parse_wiki markdown, tag_ids, group_ids, context
root = Kramdown::Document.new(markdown, input: 'GFM').root
placements = []
description_parts = []
headings = []
root.children.each_with_index do |node, index|
case node.type
when :blank
next
when :header
level = node.options.fetch(:level)
name = inline_text(node).strip
raise "#{ context }: 名前のない見出しがあります." if name.empty?
headings.pop while headings.any? && headings.last.first >= level
parent = headings.last&.last
section = { 'section_name' => name, 'description' => nil, 'children' => [] }
(parent ? parent.fetch('children') : placements) << section
headings << [level, section]
when :p
text = inline_markdown(node).strip
next if text.empty?
if headings.any?
heading = headings.last.last
heading['description'] = [heading['description'], text].compact.join("\n\n")
else
description_parts << text
end
when :ul, :ol
if headings.any? && see_also_heading?(headings.last.last.fetch('section_name'))
heading = headings.last.last
raw = original_block(markdown, root.children, index)
heading['description'] = [heading['description'], raw].compact.join("\n\n")
else
target = headings.any? ? headings.last.last.fetch('children') : placements
target.concat(node.children.map { |item| parse_list_item(item, tag_ids, group_ids, context) })
end
else
raise "#{ context }: 未対応の Markdown 構文 #{ node.type.inspect }"
end
end
[description_parts.empty? ? nil : description_parts.join("\n\n"), placements]
end
def see_also_heading? name
['see also', '関連項目'].include?(name.strip.downcase.sub(/[::]\z/, ''))
end
def original_block markdown, siblings, index
current = siblings.fetch(index)
start_line = current.options[:location]
raise 'Markdown block の位置が不明です.' unless start_line
following = siblings[(index + 1)..].find do |node|
node.options[:location] && node.options[:location] > start_line
end
end_line = following ? following.options.fetch(:location) - 1 : markdown.lines.length
markdown.lines[(start_line - 1)...end_line].join.strip
end
def parse_list_item item, tag_ids, group_ids, context
raise "#{ context }: リスト項目ではありません." if item.type != :li
nested, own = item.children.partition { |child| [:ul, :ol].include?(child.type) }
own.reject! { |child| child.type == :blank }
unless own.all? { |child| [:p, :text, :a, :codespan].include?(child.type) }
raise "#{ context }: 未対応のリスト項目があります."
end
label = own.map { |child| inline_text(child) }.join(' ').strip
raise "#{ context }: 空のリスト項目があります." if label.empty?
node =
if group_ids.key?(label)
{ 'tag_group_id' => group_ids.fetch(label) }
elsif tag_ids.key?(label)
ids = tag_ids.fetch(label)
unless ids.one?
raise "#{ context }: タグ名 #{ label.inspect } が複数の ID を指します:#{ ids.inspect }"
end
{ 'tag_id' => ids.first }
elsif label.end_with?(':', ':')
name = label.sub(/[::]\z/, '')
raise "#{ context }: 空の区分名があります." if name.empty?
if see_also_heading?(name) && nested.any?
raise "#{ context }: リスト内の See also は説明文への保管処理が未実装です."
end
{ 'section_name' => name, 'description' => nil }
else
raise "#{ context }: 解決不能なリスト項目 #{ label.inspect }"
end
node['children'] = nested.flat_map do |list|
list.children.map { |child| parse_list_item(child, tag_ids, group_ids, context) }
end
node
end
def inline_text node
case node.type
when :text, :codespan
node.value.to_s
when :softbreak, :br
' '
when :a, :p, :header, :em, :strong, :span, :li
node.children.map { |child| inline_text(child) }.join
else
raise "未対応のインライン構文 #{ node.type.inspect }"
end
end
def inline_markdown node
case node.type
when :text
node.value.to_s
when :codespan
"`#{ node.value }`"
when :softbreak
"\n"
when :br
" \n"
when :p
node.children.map { |child| inline_markdown(child) }.join
when :a
href = node.attr.fetch('href')
"[#{ node.children.map { |child| inline_markdown(child) }.join }](#{ href })"
when :em
"*#{ node.children.map { |child| inline_markdown(child) }.join }*"
when :strong
"**#{ node.children.map { |child| inline_markdown(child) }.join }**"
else
raise "未対応のインライン構文 #{ node.type.inspect }"
end
end
def validate_tree! nodes, context, depth = 1
raise "#{ context }: 配置の深さが 5 を超えてゐます." if depth > 5 && nodes.any?
nodes.each do |node|
name = node['section_name']
if name && name.length > 255
raise "#{ context }: 区分名が 255 文字を超えてゐます."
end
validate_tree!(node.fetch('children'), context, depth + 1)
end
end
def build_members nodes, date, tag_history, group_history, context
ids_of_tags = []
ids_of_groups = []
visit = -> items do
items.each do |node|
ids_of_tags << node['tag_id'] if node.key?('tag_id')
ids_of_groups << node['tag_group_id'] if node.key?('tag_group_id')
visit.(node.fetch('children'))
end
end
visit.(nodes)
tags = ids_of_tags.uniq.map do |id|
{ 'tag_id' => id,
'version_no' => version_at(tag_history[id], date, context, "Tag ##{ id }") }
end
groups = ids_of_groups.uniq.map do |id|
{ 'tag_group_id' => id,
'version_no' => version_at(group_history[id], date, context, "TagGroup ##{ id }") }
end
tags + groups
end
def version_at history, date, context, target
raise "#{ context }: #{ target } の履歴が存在しません." unless history&.any?
timestamp = Time.parse(date.to_s)
candidates = history.select do |row|
Time.parse(row.fetch('created_at').to_s) <= timestamp
end
if candidates.empty?
raise "#{ context }: #{ target } は #{ date } 時点で存在しません."
end
candidates.max_by { |row|
[Time.parse(row.fetch('created_at').to_s), row.fetch('version_no').to_i]
}.fetch('version_no').to_i
end
def validate_current_groups! prepared
edges = { }
prepared.each do |group_id, versions|
edges[group_id] = versions.last.fetch(:members)
.filter_map { |member| member['tag_group_id'] }
end
visiting = { }
visited = { }
visit = -> group_id do
raise "Tag Group の循環参照を検出しました:#{ group_id }" if visiting[group_id]
return if visited[group_id]
visiting[group_id] = true
edges.fetch(group_id, []).each { |child_id| visit.(child_id) }
visiting.delete(group_id)
visited[group_id] = true
end
edges.keys.each { |group_id| visit.(group_id) }
end
def quote(value) = connection.quote(value)
def insert_row table, values
columns = values.keys.join(', ')
payload = values.values.map { |value| quote(value) }.join(', ')
execute "INSERT INTO #{ table } (#{ columns }) VALUES (#{ payload })"
end
end
+9
ファイルの表示
@@ -0,0 +1,9 @@
class DropWikiPages < ActiveRecord::Migration[8.0]
def change
drop_table :wiki_versions
drop_table :wiki_revision_lines
drop_table :wiki_lines
drop_table :wiki_revisions
drop_table :wiki_pages
end
end
+11
ファイルの表示
@@ -0,0 +1,11 @@
class RemoveTagNameIdFromTags < ActiveRecord::Migration[8.0]
def up
remove_foreign_key :tags, column: :tag_name_id
remove_foreign_key :tag_names, column: :canonical_id
remove_column :tags, :tag_name_id
remove_column :tag_names, :canonical_id
end
def down = raise ActiveRecord::IrreversibleMigration, '戻せません.'
end
+15
ファイルの表示
@@ -0,0 +1,15 @@
class DeleteNameForWikiFromTagNames < ActiveRecord::Migration[8.0]
def up
execute <<~SQL
DELETE
FROM
tag_names
WHERE
tag_id IS NULL
SQL
change_column_null :tag_names, :tag_id, false
end
def down = raise ActiveRecord::IrreversibleMigration, '戻せません.'
end
生成ファイル
+90 -89
ファイルの表示
@@ -10,7 +10,7 @@
#
# It's strongly recommended that you check this file into your version control system.
ActiveRecord::Schema[8.0].define(version: 2026_10_05_000000) do
ActiveRecord::Schema[8.0].define(version: 2026_10_15_010000) do
create_table "active_storage_attachments", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.string "name", null: false
t.string "record_type", null: false
@@ -483,6 +483,80 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_05_000000) do
t.check_constraint "`version_no` > 0", name: "chk_tag_descriptions_version_no_positive"
end
create_table "tag_group_groups", primary_key: ["tag_group_id", "child_tag_group_id"], charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.bigint "tag_group_id", null: false
t.bigint "child_tag_group_id", null: false
t.bigint "created_by_user_id"
t.datetime "created_at", null: false
t.index ["child_tag_group_id"], name: "index_tag_group_groups_on_child_tag_group_id"
t.index ["created_by_user_id"], name: "index_tag_group_groups_on_created_by_user_id"
t.check_constraint "`tag_group_id` <> `child_tag_group_id`", name: "chk_tag_group_groups_tag_group_id_not_child_tag_group_id"
end
create_table "tag_group_placements", primary_key: ["tag_group_id", "position"], charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.bigint "tag_group_id", null: false
t.integer "position", null: false
t.integer "parent_position"
t.string "section_name"
t.text "description"
t.bigint "tag_id"
t.bigint "child_tag_group_id"
t.bigint "created_by_user_id"
t.bigint "updated_by_user_id"
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
t.index ["child_tag_group_id"], name: "index_tag_group_placements_on_child_tag_group_id"
t.index ["created_by_user_id"], name: "index_tag_group_placements_on_created_by_user_id"
t.index ["tag_group_id", "child_tag_group_id"], name: "fk_rails_7342b54ced"
t.index ["tag_group_id", "parent_position"], name: "fk_rails_3af6ee71ca"
t.index ["tag_group_id", "tag_id"], name: "fk_rails_365dc60759"
t.index ["tag_id"], name: "index_tag_group_placements_on_tag_id"
t.index ["updated_by_user_id"], name: "index_tag_group_placements_on_updated_by_user_id"
t.check_constraint "(((`section_name` is not null) and (`tag_id` is null) and (`child_tag_group_id` is null)) or ((`section_name` is null) and (`tag_id` is not null) and (`child_tag_group_id` is null)) or ((`section_name` is null) and (`tag_id` is null) and (`child_tag_group_id` is not null))) and ((`section_name` is not null) or (`description` is null))", name: "chk_tag_group_placements_references_exclusive"
t.check_constraint "`position` > `parent_position`", name: "chk_tag_group_placements_position_gt_parent_position"
end
create_table "tag_group_tags", primary_key: ["tag_group_id", "tag_id"], charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.bigint "tag_group_id", null: false
t.bigint "tag_id", null: false
t.bigint "created_by_user_id"
t.datetime "created_at", null: false
t.index ["created_by_user_id"], name: "index_tag_group_tags_on_created_by_user_id"
t.index ["tag_id"], name: "index_tag_group_tags_on_tag_id"
end
create_table "tag_group_versions", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.bigint "tag_group_id", null: false
t.integer "version_no", null: false
t.string "event_type", null: false
t.string "name", null: false
t.text "description"
t.json "members", null: false
t.json "placements", null: false
t.bigint "created_by_user_id"
t.datetime "created_at", null: false
t.index ["created_by_user_id"], name: "index_tag_group_versions_on_created_by_user_id"
t.index ["tag_group_id", "version_no"], name: "index_tag_group_versions_on_tag_group_id_and_version_no", unique: true
t.index ["tag_group_id"], name: "index_tag_group_versions_on_tag_group_id"
t.check_constraint "`version_no` > 0", name: "chk_tag_group_versions_version_no_positive"
t.check_constraint "json_schema_valid(_utf8mb4'{\"type\":\"array\",\"items\":{\"type\":\"object\",\"required\":[\"children\"],\"properties\":{\"section_name\":{\"type\":\"string\"},\"description\":{\"type\":[\"string\",\"null\"]},\"tag_id\":{\"type\":\"integer\",\"minimum\":1},\"tag_group_id\":{\"type\":\"integer\",\"minimum\":1},\"children\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"required\":[\"children\"],\"properties\":{\"section_name\":{\"type\":\"string\"},\"description\":{\"type\":[\"string\",\"null\"]},\"tag_id\":{\"type\":\"integer\",\"minimum\":1},\"tag_group_id\":{\"type\":\"integer\",\"minimum\":1},\"children\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"required\":[\"children\"],\"properties\":{\"section_name\":{\"type\":\"string\"},\"description\":{\"type\":[\"string\",\"null\"]},\"tag_id\":{\"type\":\"integer\",\"minimum\":1},\"tag_group_id\":{\"type\":\"integer\",\"minimum\":1},\"children\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"required\":[\"children\"],\"properties\":{\"section_name\":{\"type\":\"string\"},\"description\":{\"type\":[\"string\",\"null\"]},\"tag_id\":{\"type\":\"integer\",\"minimum\":1},\"tag_group_id\":{\"type\":\"integer\",\"minimum\":1},\"children\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"required\":[\"children\"],\"properties\":{\"section_name\":{\"type\":\"string\"},\"description\":{\"type\":[\"string\",\"null\"]},\"tag_id\":{\"type\":\"integer\",\"minimum\":1},\"tag_group_id\":{\"type\":\"integer\",\"minimum\":1},\"children\":{\"type\":\"array\",\"maxItems\":0}},\"oneOf\":[{\"required\":[\"section_name\"]},{\"required\":[\"tag_id\"]},{\"required\":[\"tag_group_id\"]}],\"dependencies\":{\"description\":[\"section_name\"]},\"additionalProperties\":false}}},\"oneOf\":[{\"required\":[\"section_name\"]},{\"required\":[\"tag_id\"]},{\"required\":[\"tag_group_id\"]}],\"dependencies\":{\"description\":[\"section_name\"]},\"additionalProperties\":false}}},\"oneOf\":[{\"required\":[\"section_name\"]},{\"required\":[\"tag_id\"]},{\"required\":[\"tag_group_id\"]}],\"dependencies\":{\"description\":[\"section_name\"]},\"additionalProperties\":false}}},\"oneOf\":[{\"required\":[\"section_name\"]},{\"required\":[\"tag_id\"]},{\"required\":[\"tag_group_id\"]}],\"dependencies\":{\"description\":[\"section_name\"]},\"additionalProperties\":false}}},\"oneOf\":[{\"required\":[\"section_name\"]},{\"required\":[\"tag_id\"]},{\"required\":[\"tag_group_id\"]}],\"dependencies\":{\"description\":[\"section_name\"]},\"additionalProperties\":false}}',`placements`)", name: "chk_tag_group_versions_placements_schema"
t.check_constraint "json_schema_valid(_utf8mb4'{\"type\":\"array\",\"uniqueItems\":true,\"items\":{\"oneOf\":[{\"type\":\"object\",\"required\":[\"tag_id\",\"version_no\"],\"properties\":{\"tag_id\":{\"type\":\"integer\",\"minimum\":1},\"version_no\":{\"type\":\"integer\",\"minimum\":1}},\"additionalProperties\":false},{\"type\":\"object\",\"required\":[\"tag_group_id\",\"version_no\"],\"properties\":{\"tag_group_id\":{\"type\":\"integer\",\"minimum\":1},\"version_no\":{\"type\":\"integer\",\"minimum\":1}},\"additionalProperties\":false}]}}',`members`)", name: "chk_tag_group_versions_members_schema"
end
create_table "tag_groups", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.integer "version_no", null: false
t.string "name", null: false
t.text "description"
t.bigint "created_by_user_id"
t.bigint "updated_by_user_id"
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
t.index ["created_by_user_id"], name: "index_tag_groups_on_created_by_user_id"
t.index ["name"], name: "index_tag_groups_on_name", unique: true
t.index ["updated_by_user_id"], name: "index_tag_groups_on_updated_by_user_id"
t.check_constraint "`version_no` > 0", name: "chk_tag_groups_version_no_positive"
end
create_table "tag_implications", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.bigint "tag_id", null: false
t.bigint "parent_tag_id", null: false
@@ -504,17 +578,15 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_05_000000) do
end
create_table "tag_names", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.bigint "tag_id"
t.bigint "tag_id", null: false
t.string "language_code", limit: 16, null: false
t.string "name", null: false
t.string "script_code", limit: 4, null: false
t.boolean "primary_flg", null: false
t.boolean "auto_generated", default: false, null: false
t.bigint "canonical_id"
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
t.virtual "primary_tag_id", type: :bigint, as: "(case when `primary_flg` then `tag_id` else NULL end)"
t.index ["canonical_id"], name: "index_tag_names_on_canonical_id"
t.index ["language_code", "name"], name: "index_tag_names_on_language_code_and_name", unique: true
t.index ["primary_tag_id", "language_code"], name: "index_tag_names_on_primary_tag_id_and_language_code", unique: true
t.index ["script_code"], name: "fk_rails_dd783b3d1c"
@@ -551,7 +623,6 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_05_000000) do
end
create_table "tags", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.bigint "tag_name_id", null: false
t.string "category", default: "general", null: false
t.datetime "deprecated_at"
t.datetime "created_at", null: false
@@ -559,7 +630,6 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_05_000000) do
t.integer "post_count", default: 0, null: false
t.integer "version_no", null: false
t.index ["deprecated_at"], name: "index_tags_on_deprecated_at"
t.index ["tag_name_id"], name: "index_tags_on_tag_name_id", unique: true
t.check_constraint "(`deprecated_at` is null) or (`category` <> _utf8mb4'nico')", name: "chk_tags_deprecated_at_not_nico"
t.check_constraint "`version_no` > 0", name: "chk_tags_version_no_positive"
end
@@ -699,76 +769,6 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_05_000000) do
t.index ["banned_at"], name: "index_users_on_banned_at"
end
create_table "wiki_lines", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.string "sha256", limit: 64, null: false
t.text "body", null: false
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
t.index ["sha256"], name: "index_wiki_lines_on_sha256", unique: true
end
create_table "wiki_pages", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.bigint "tag_name_id", null: false
t.text "body", null: false
t.bigint "created_user_id", null: false
t.bigint "updated_user_id", null: false
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
t.datetime "discarded_at"
t.integer "version_no", null: false
t.index ["created_user_id"], name: "index_wiki_pages_on_created_user_id"
t.index ["discarded_at"], name: "index_wiki_pages_on_discarded_at"
t.index ["tag_name_id"], name: "index_wiki_pages_on_tag_name_id", unique: true
t.index ["updated_user_id"], name: "index_wiki_pages_on_updated_user_id"
t.check_constraint "`version_no` > 0", name: "chk_wiki_pages_version_no_positive"
end
create_table "wiki_revision_lines", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.bigint "wiki_revision_id", null: false
t.integer "position", null: false
t.bigint "wiki_line_id", null: false
t.index ["wiki_line_id"], name: "index_wiki_revision_lines_on_wiki_line_id"
t.index ["wiki_revision_id", "position"], name: "index_wiki_revision_lines_on_wiki_revision_id_and_position", unique: true
t.index ["wiki_revision_id", "wiki_line_id"], name: "index_wiki_revision_lines_on_wiki_revision_id_and_wiki_line_id"
t.index ["wiki_revision_id"], name: "index_wiki_revision_lines_on_wiki_revision_id"
end
create_table "wiki_revisions", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.bigint "wiki_page_id", null: false
t.bigint "base_revision_id"
t.bigint "created_user_id", null: false
t.integer "kind", default: 0, null: false
t.bigint "redirect_page_id"
t.string "message"
t.integer "lines_count", default: 0, null: false
t.string "tree_sha256", limit: 64
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
t.index ["base_revision_id"], name: "index_wiki_revisions_on_base_revision_id"
t.index ["created_user_id"], name: "index_wiki_revisions_on_created_user_id"
t.index ["kind"], name: "index_wiki_revisions_on_kind"
t.index ["redirect_page_id"], name: "index_wiki_revisions_on_redirect_page_id"
t.index ["tree_sha256"], name: "index_wiki_revisions_on_tree_sha256"
t.index ["wiki_page_id", "id"], name: "index_wiki_revisions_on_wiki_page_id_and_id"
t.index ["wiki_page_id"], name: "index_wiki_revisions_on_wiki_page_id"
end
create_table "wiki_versions", charset: "utf8mb4", collation: "utf8mb4_0900_ai_ci", force: :cascade do |t|
t.bigint "wiki_page_id", null: false
t.integer "version_no", null: false
t.string "event_type", null: false
t.string "title", null: false
t.text "body", null: false
t.text "reason"
t.datetime "created_at", null: false
t.bigint "created_by_user_id"
t.index ["created_by_user_id"], name: "index_wiki_versions_on_created_by_user_id"
t.index ["wiki_page_id", "version_no"], name: "index_wiki_versions_on_wiki_page_id_and_version_no", unique: true
t.index ["wiki_page_id"], name: "index_wiki_versions_on_wiki_page_id"
t.check_constraint "`event_type` in (_utf8mb4'create',_utf8mb4'update',_utf8mb4'discard',_utf8mb4'restore')", name: "wiki_versions_event_type_valid"
t.check_constraint "`version_no` > 0", name: "wiki_versions_version_no_positive"
end
add_foreign_key "active_storage_attachments", "active_storage_blobs", column: "blob_id"
add_foreign_key "active_storage_variant_records", "active_storage_blobs", column: "blob_id"
add_foreign_key "gekanator_ai_runs", "gekanator_question_suggestions"
@@ -823,16 +823,28 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_05_000000) do
add_foreign_key "tag_descriptions", "tags"
add_foreign_key "tag_descriptions", "users", column: "created_by_user_id"
add_foreign_key "tag_descriptions", "users", column: "updated_by_user_id"
add_foreign_key "tag_group_groups", "tag_groups"
add_foreign_key "tag_group_groups", "tag_groups", column: "child_tag_group_id"
add_foreign_key "tag_group_groups", "users", column: "created_by_user_id"
add_foreign_key "tag_group_placements", "tag_group_groups", column: ["tag_group_id", "child_tag_group_id"], primary_key: ["tag_group_id", "child_tag_group_id"]
add_foreign_key "tag_group_placements", "tag_group_placements", column: ["tag_group_id", "parent_position"], primary_key: ["tag_group_id", "position"]
add_foreign_key "tag_group_placements", "tag_group_tags", column: ["tag_group_id", "tag_id"], primary_key: ["tag_group_id", "tag_id"]
add_foreign_key "tag_group_placements", "tag_groups"
add_foreign_key "tag_group_placements", "users", column: "created_by_user_id"
add_foreign_key "tag_group_placements", "users", column: "updated_by_user_id"
add_foreign_key "tag_group_tags", "tag_groups"
add_foreign_key "tag_group_tags", "tags"
add_foreign_key "tag_group_tags", "users", column: "created_by_user_id"
add_foreign_key "tag_groups", "users", column: "created_by_user_id"
add_foreign_key "tag_groups", "users", column: "updated_by_user_id"
add_foreign_key "tag_implications", "tags"
add_foreign_key "tag_implications", "tags", column: "parent_tag_id"
add_foreign_key "tag_names", "languages", column: "language_code", primary_key: "code"
add_foreign_key "tag_names", "scripts", column: "script_code", primary_key: "code"
add_foreign_key "tag_names", "tag_names", column: "canonical_id"
add_foreign_key "tag_names", "tags"
add_foreign_key "tag_similarities", "tags"
add_foreign_key "tag_similarities", "tags", column: "target_tag_id"
add_foreign_key "tag_versions", "users", column: "created_by_user_id"
add_foreign_key "tags", "tag_names"
add_foreign_key "theatre_comments", "theatres"
add_foreign_key "theatre_comments", "users"
add_foreign_key "theatre_programmes", "posts"
@@ -857,15 +869,4 @@ ActiveRecord::Schema[8.0].define(version: 2026_10_05_000000) do
add_foreign_key "user_post_views", "posts"
add_foreign_key "user_post_views", "users"
add_foreign_key "user_theme_slots", "users"
add_foreign_key "wiki_pages", "tag_names"
add_foreign_key "wiki_pages", "users", column: "created_user_id"
add_foreign_key "wiki_pages", "users", column: "updated_user_id"
add_foreign_key "wiki_revision_lines", "wiki_lines"
add_foreign_key "wiki_revision_lines", "wiki_revisions"
add_foreign_key "wiki_revisions", "users", column: "created_user_id"
add_foreign_key "wiki_revisions", "wiki_pages"
add_foreign_key "wiki_revisions", "wiki_pages", column: "redirect_page_id"
add_foreign_key "wiki_revisions", "wiki_revisions", column: "base_revision_id"
add_foreign_key "wiki_versions", "users", column: "created_by_user_id"
add_foreign_key "wiki_versions", "wiki_pages"
end
+5
ファイルの表示
@@ -8,6 +8,11 @@
# MovieGenre.find_or_create_by!(name: genre_name)
# end
ApplicationRecord.transaction do
Language.find_or_create_by!(code: 'ain') { _1.name = 'ain' }
Script.find_or_create_by!(code: 'Kana') { _1.name = 'Kana' }
end
post_url_sanitisation_rules = [
{ priority: 10,
source_pattern: '\Ahttps?://youtu\.be/([^/?#]+)(?:[?#].*)?\z',
+3 -3
ファイルの表示
@@ -3,8 +3,8 @@ namespace :nico do
task export: :environment do
require 'open3'
mysql_user = ENV.fetch('MYSQL_USER')
mysql_pass = ENV.fetch('MYSQL_PASS')
mysql_user = ENV.fetch('NIZIKA_NICO_MYSQL_USER')
mysql_pass = ENV.fetch('NIZIKA_NICO_MYSQL_PASS')
nizika_nico_path = ENV.fetch('NIZIKA_NICO_PATH')
videos = Post.where('url LIKE ?', '%nicovideo.jp/watch/%').pluck(:url).filter_map {
@@ -14,7 +14,7 @@ namespace :nico do
next if videos.empty?
_, stderr, status = Open3.capture3(
{ 'MYSQL_USER' => mysql_user, 'MYSQL_PASS' => mysql_pass },
{ 'NIZIKA_NICO_MYSQL_USER' => mysql_user, 'NIZIKA_NICO_MYSQL_PASS' => mysql_pass },
'python3', '-m', 'tracked_videos.put_bulk_upsert', *videos,
chdir: nizika_nico_path)
+51
ファイルの表示
@@ -0,0 +1,51 @@
namespace :help do
desc 'ヘルプ移行'
task migrate: :environment do
help_path = ENV.fetch('BTRC_HUB_HELP_PATH')
abort 'ヘルプ置場指定してね.' if help_path.blank?
# ファイルがすでに存在する場合は通さなぃ.
abort 'ファイル消してね.' if Dir.glob("#{ help_path }/*").present?
# 現状物理削除はなぃはず
pages =
WikiPage
.joins(:tag_name)
.where(tag_names: { tag_id: nil, language_code: 'ja' })
.where.not("tag_names.name LIKE 'タググループ:%'")
puts "移行対象: #{pages.count} 件"
abort '移行対象がありません.' if pages.empty?
pages.find_each do |page|
puts page.body
puts page.title
puts 'この identifier は何にする?'
identifier = $stdin.gets.chomp
Dir.mkdir "#{ help_path }/#{ identifier }"
Dir.mkdir "#{ help_path }/#{ identifier }/editions"
Dir.mkdir "#{ help_path }/#{ identifier }/editions/ja"
last_file_name = ''
WikiVersion.where(wiki_page_id: page.id).order(:version_no).each do |ver|
last_file_name, last_file_name_no = last_file_name.split('.')
file_name = ver.created_at.strftime('%Y-%m-%d')
if file_name == last_file_name
file_name += ".#{ (last_file_name_no || 1).to_i + 1 }"
end
body = "# #{ ver.title.sub(/^ヘルプ:/, '') }\n\n#{ ver.body }"
File.write "#{ help_path }/#{ identifier }/editions/ja/#{ file_name }.mdx", body
last_file_name = file_name
end
article_meta = { 'identifier' => identifier,
'canonical_language' => 'ja',
'languages' => { 'ja' => { 'current_edition' => last_file_name } } }
File.write "#{ help_path }/#{ identifier }/article.yml", article_meta.to_yaml
end
end
end
+3 -3
ファイルの表示
@@ -54,11 +54,11 @@ namespace :nico do
.find_each(&:destroy!)
end
mysql_user = ENV['MYSQL_USER']
mysql_pass = ENV['MYSQL_PASS']
mysql_user = ENV['NIZIKA_NICO_MYSQL_USER']
mysql_pass = ENV['NIZIKA_NICO_MYSQL_PASS']
nizika_nico_path = ENV['NIZIKA_NICO_PATH']
stdout, stderr, status = Open3.capture3(
{ 'MYSQL_USER' => mysql_user, 'MYSQL_PASS' => mysql_pass },
{ 'NIZIKA_NICO_MYSQL_USER' => mysql_user, 'NIZIKA_NICO_MYSQL_PASS' => mysql_pass },
'python3', "#{ nizika_nico_path }/get_videos.py")
unless status.success?
-34
ファイルの表示
@@ -1,34 +0,0 @@
require 'rails_helper'
require_relative '../../db/migrate/20261003010000_migrate_tag_descriptions_from_wiki_pages'
RSpec.describe 'Tag description migration identity' do
it 'preserves every revision ID and repeated body in an independent sequence' do
user = create_member_user!
tag_name = create(:tag_name, name: 'description_migration')
page = Wiki::Commit.create_content!(
tag_name:, body: 'A', created_by_user: user)
revisions = [page.current_revision]
['A', 'B', 'B', 'A'].each do |body|
revisions << Wiki::Commit.content!(
page:, body:, created_user: user, base_revision_id: revisions.last.id)
end
tag = create(:tag, primary_tag_name: tag_name)
legacy_state = [page.reload.attributes, revisions.map(&:attributes)]
expect(page.wiki_versions.count).to eq(3)
migration = MigrateTagDescriptionsFromWikiPages.new
migration.suppress_messages { migration.up }
versions = TagDescriptionVersion.where(tag_id: tag.id, language_code: 'ja')
.order(:version_no)
expect(versions.pluck(:id)).to eq(revisions.map(&:id))
expect(versions.pluck(:version_no)).to eq([1, 2, 3, 4, 5])
expect(versions.pluck(:body)).to eq(['A', 'A', 'B', 'B', 'A'])
expect(versions.pluck(:created_by_user_id)).to eq([user.id] * 5)
expect(versions.pluck(:created_at)).to eq(revisions.map(&:created_at))
expect(TagDescription.find(page.id)).to have_attributes(
tag_id: tag.id, language_code: 'ja', version_no: 5, body: 'A')
expect([page.reload.attributes, revisions.map { |revision| revision.reload.attributes }])
.to eq(legacy_state)
end
end
+110
ファイルの表示
@@ -0,0 +1,110 @@
require 'rails_helper'
RSpec.describe 'Tag group persistence constraints' do
include TagGroupRecords
let!(:user) { create_member_user! }
let!(:tag) { group_tag('制約確認') }
let!(:group_id) do
seed_group(
name: '制約確認',
members: [{ 'tag_id' => tag.id, 'version_no' => tag.version_no }])
end
def placement attributes
insert_group_row('tag_group_placements', {
tag_group_id: group_id, position: 1,
created_at: Time.current, updated_at: Time.current }.merge(attributes))
end
def reject_statement
before_state = group_state
expect { yield }.to raise_error(ActiveRecord::StatementInvalid)
expect(group_state).to eq(before_state)
end
it 'allows membership without placement' do
expect(group_rows('tag_group_tags', tag_group_id: group_id).size).to eq(1)
expect(group_rows('tag_group_placements', tag_group_id: group_id)).to be_empty
end
it 'allows repeated Tag placements with unique membership' do
placement(tag_id: tag.id)
placement(tag_id: tag.id, position: 2)
expect(group_rows('tag_group_tags', tag_group_id: group_id).size).to eq(1)
expect(group_rows('tag_group_placements', tag_group_id: group_id).size).to eq(2)
end
it 'rejects duplicate Tag membership' do
reject_statement do
insert_group_row('tag_group_tags', {
tag_group_id: group_id, tag_id: tag.id, created_at: Time.current })
end
end
it 'rejects duplicate child membership' do
child_id = seed_group(name: '子')
attributes = { tag_group_id: group_id, child_tag_group_id: child_id,
created_at: Time.current }
insert_group_row('tag_group_groups', attributes)
reject_statement { insert_group_row('tag_group_groups', attributes) }
end
it 'rejects direct self membership' do
reject_statement do
insert_group_row('tag_group_groups', {
tag_group_id: group_id, child_tag_group_id: group_id, created_at: Time.current })
end
end
it 'rejects a placement without a kind' do
reject_statement { placement({ }) }
end
it 'rejects simultaneous Section and Tag kinds' do
reject_statement { placement(section_name: '区分', tag_id: tag.id) }
end
it 'rejects a description attached to a Tag placement' do
reject_statement { placement(tag_id: tag.id, description: '不正') }
end
it 'allows an optional description on a Section' do
placement(section_name: '区分', description: '説明')
expect(group_rows('tag_group_placements', tag_group_id: group_id).sole)
.to include('section_name' => '区分', 'description' => '説明')
end
it 'rejects a Tag placement absent from membership' do
other = group_tag('非メンバー')
reject_statement { placement(tag_id: other.id) }
end
it 'rejects a missing parent placement' do
reject_statement { placement(section_name: '子', parent_position: 9) }
end
it 'rejects a child ordered before its parent' do
placement(section_name: '親', position: 2)
reject_statement { placement(section_name: '子', position: 1, parent_position: 2) }
end
it 'rejects duplicate group names' do
reject_statement { seed_group(name: '制約確認') }
end
it 'rejects a missing mandatory group name' do
reject_statement { seed_group(name: nil) }
end
it 'rejects a six-level snapshot' do
nodes = []
6.times { nodes = [{ 'tag_id' => tag.id, 'children' => nodes }] }
reject_statement do
insert_group_row('tag_group_versions', {
tag_group_id: group_id, version_no: 2, event_type: 'update', name: '制約確認',
members: [{ 'tag_id' => tag.id, 'version_no' => tag.version_no }].to_json,
placements: nodes.to_json, created_at: Time.current })
end
end
end
+16 -5
ファイルの表示
@@ -1,11 +1,17 @@
require 'rails_helper'
RSpec.describe 'TagName database invariants' do
def insert_name!(**attributes)
TagName.insert_all!([{ name: 'constraint_name', language_code: 'ja',
script_code: 'Jpan', primary_flg: true,
created_at: Time.current, updated_at: Time.current,
**attributes }])
def insert_name! **attributes
defaults = {
tag_id: create(:tag, primary_tag_name: nil).id,
name: 'constraint_name',
language_code: 'ja',
script_code: 'Jpan',
primary_flg: true,
created_at: Time.current,
updated_at: Time.current }
TagName.insert_all!([defaults.merge(attributes)])
end
it 'defaults persisted names to human-confirmed provenance' do
@@ -33,4 +39,9 @@ RSpec.describe 'TagName database invariants' do
.to raise_error(ActiveRecord::RecordNotUnique)
expect(TagName.where(tag:, language_code: 'ja', primary_flg: true).count).to eq(1)
end
it 'rejects a name without a tag' do
expect { insert_name!(tag_id: nil) }
.to raise_error(ActiveRecord::NotNullViolation)
end
end
+23
ファイルの表示
@@ -0,0 +1,23 @@
require 'rails_helper'
RSpec.describe 'Schema after Wiki removal' do
it 'has no legacy Wiki tables or name indirection columns' do
connection = ApplicationRecord.connection
['wiki_pages', 'wiki_revisions', 'wiki_lines', 'wiki_revision_lines', 'wiki_versions']
.each do |table|
expect(connection.table_exists?(table)).to be(false)
end
expect(connection.column_exists?(:tags, :tag_name_id)).to be(false)
expect(connection.column_exists?(:tag_names, :canonical_id)).to be(false)
end
it 'requires TagName ownership while keeping the native resources and history' do
connection = ApplicationRecord.connection
expect(TagName.columns_hash.fetch('tag_id').null).to be(false)
expect(connection.foreign_key_exists?(:tag_names, :tags)).to be(true)
['tag_descriptions', 'tag_description_versions', 'tag_groups', 'tag_group_versions']
.each do |table|
expect(connection.table_exists?(table)).to be(true)
end
end
end
+2
ファイルの表示
@@ -6,6 +6,8 @@ FactoryBot.define do
auto_generated { false }
sequence(:name) { |number| "tag-#{ SecureRandom.hex(4) }-#{ number }" }
association :tag, primary_tag_name: nil
trait :alias do
primary_flg { false }
end
+7 -18
ファイルの表示
@@ -4,7 +4,7 @@ FactoryBot.define do
primary_name { nil }
primary_tag_name do
attributes = primary_name.nil? ? { } : { name: primary_name }
build(:tag_name, **attributes)
build(:tag_name, tag: nil, **attributes)
end
end
@@ -12,30 +12,19 @@ FactoryBot.define do
post_count { 0 }
after(:build) do |tag, evaluator|
next unless evaluator.primary_tag_name
evaluator.primary_tag_name.primary_flg = true
evaluator.primary_tag_name.tag = tag
tag.tag_names.target << evaluator.primary_tag_name
end
before(:create) do |tag, evaluator|
evaluator.primary_tag_name.save!
# Compatibility write only; ownership is established through tag_id below.
tag.tag_name_id = evaluator.primary_tag_name.id
end
after(:create) do |tag, evaluator|
next unless evaluator.primary_tag_name
evaluator.primary_tag_name.update!(tag:, primary_flg: true)
Locale.where(deprecated_at: nil).order(:language_code, :created_at, :script_code).to_a
.uniq(&:language_code).each do |locale|
next if tag.tag_names.exists?(language_code: locale.language_code,
primary_flg: true)
create(:tag_name,
tag:, language_code: locale.language_code,
script_code: locale.script_code,
name: TagName.generate_name(locale, tag, evaluator.primary_tag_name),
primary_flg: true, auto_generated: true)
end
TagNameCompletion.for_tag!(tag)
end
end
end
-9
ファイルの表示
@@ -1,9 +0,0 @@
FactoryBot.define do
factory :wiki_page do
title { "TestPage" }
association :created_user, factory: :user
association :updated_user, factory: :user
body { ' ' }
end
end
+2 -3
ファイルの表示
@@ -6,9 +6,8 @@ RSpec.describe Language, type: :model do
expect(described_class.where(deprecated_at: nil).pluck(:code)).to eq(['ja'])
end
it 'accepts English when multilingual support is enabled' do
pending '日本語以外の Language 有効化は別課題で実装する'
expect { described_class.create!(code: 'en', name: 'English') }
it 'accepts Ainu when multilingual support is enabled' do
expect { described_class.create!(code: 'ain', name: 'ain') }
.to change(described_class, :count).by(1)
end
+82
ファイルの表示
@@ -0,0 +1,82 @@
require 'rails_helper'
RSpec.describe 'Locale activation', type: :model do
let!(:language) { Language.create!(code: 'ain', name: 'ain') }
let!(:script) { Script.create!(code: 'Kana', name: 'Kana') }
let!(:first) { create(:tag, primary_name: 'activation_first') }
let!(:second) { create(:tag, primary_name: 'activation_second') }
def build_locale deprecated_at: nil
Locale.new(code: 'ain-Kana', language_code: language.code,
script_code: script.code, region_code: 392, variants: '',
name: 'ain-Kana', deprecated_at:)
end
def generated_names
TagName.where(language_code: 'ain', primary_flg: true)
end
def fail_second_generation!
allow(TagName).to receive(:generate_name).and_call_original
allow(TagName).to receive(:generate_name)
.with(kind_of(Locale), second, second.tag_name('ja'))
.and_raise('backfill failed')
end
it 'rolls back a newly created locale and the preceding generated names' do
locale = build_locale
fail_second_generation!
expect { locale.save! }.to raise_error('backfill failed')
expect(Locale.exists?(code: 'ain-Kana')).to be(false)
expect(generated_names).to be_empty
expect(first.reload.name('ja')).to eq('activation_first')
expect(second.reload.name('ja')).to eq('activation_second')
end
it 'does not backfill while the locale is inactive' do
build_locale(deprecated_at: Time.current).save!
expect(generated_names).to be_empty
end
it 'backfills missing primary names when the locale becomes active' do
locale = build_locale(deprecated_at: Time.current)
locale.save!
locale.update!(deprecated_at: nil)
expect(generated_names.pluck(:tag_id)).to contain_exactly(first.id, second.id)
expect(generated_names).to all(have_attributes(
script_code: 'Kana', primary_flg: true, auto_generated: true))
end
it 'rolls back activation and all generated names when backfill fails' do
locale = build_locale(deprecated_at: Time.current)
locale.save!
previous_deprecated_at = locale.reload.deprecated_at
fail_second_generation!
expect { locale.update!(deprecated_at: nil) }.to raise_error('backfill failed')
expect(locale.reload.deprecated_at).to eq(previous_deprecated_at)
expect(generated_names).to be_empty
end
it 'preserves a human primary through deactivation and repeated activation' do
locale = build_locale
locale.save!
primary = generated_names.find_by!(tag: first)
primary.update!(name: 'human_confirmed', auto_generated: false)
snapshot = generated_names.order(:id).map(&:attributes)
2.times do
locale.update!(deprecated_at: Time.current)
locale.update!(deprecated_at: nil)
end
expect(generated_names.order(:id).map(&:attributes)).to eq(snapshot)
expect(primary.reload).to have_attributes(name: 'human_confirmed', auto_generated: false)
end
end
+111 -55
ファイルの表示
@@ -1,13 +1,13 @@
require 'rails_helper'
RSpec.describe Locale, type: :model do
include_context 'English locale'
include_context 'Ainu locale'
def create_english_locale!
prepare_english_reference!
described_class.create!(code: 'en', language_code: 'en',
script_code: 'Latn', region_code: 826,
variants: '', name: 'English')
def create_ainu_locale!
prepare_ainu_reference!
described_class.create!(code: 'ain-Kana', language_code: 'ain',
script_code: 'Kana', region_code: 392,
variants: '', name: 'Ainu')
end
it 'keeps the active Nipponese locale consistent with its identity tuple' do
@@ -27,7 +27,6 @@ RSpec.describe Locale, type: :model do
end
it 'generates a primary name for every existing tag using the name generator' do
pending '日本語以外の Locale 有効化は別課題で実装する'
first = create(:tag, primary_name: 'first_existing_tag')
second = create(:tag, primary_name: 'second_existing_tag')
allow(TagName).to receive(:generate_name).and_call_original
@@ -35,95 +34,152 @@ RSpec.describe Locale, type: :model do
.with(kind_of(described_class), first, primary_tag_name_for(first, 'ja'))
.and_return('name_from_generator')
locale = create_english_locale!
locale = create_ainu_locale!
expect(TagName).to have_received(:generate_name)
.with(locale, first, primary_tag_name_for(first, 'ja'))
expect(TagName).to have_received(:generate_name)
.with(locale, second, primary_tag_name_for(second, 'ja'))
expect(first.tag_names.find_by!(language_code: 'en', primary_flg: true))
expect(first.tag_names.find_by!(language_code: 'ain', primary_flg: true))
.to have_attributes(name: 'name_from_generator', tag_id: first.id,
language_code: 'en', script_code: 'Latn',
language_code: 'ain', script_code: 'Kana',
primary_flg: true, auto_generated: true)
expect(second.tag_names.find_by!(language_code: 'en', primary_flg: true))
.to have_attributes(tag_id: second.id, language_code: 'en',
script_code: 'Latn', primary_flg: true,
expect(second.tag_names.find_by!(language_code: 'ain', primary_flg: true))
.to have_attributes(tag_id: second.id, language_code: 'ain',
script_code: 'Kana', primary_flg: true,
auto_generated: true)
expect(TagName.where(language_code: 'en', primary_flg: true,
expect(TagName.where(language_code: 'ain', primary_flg: true,
tag_id: [first.id, second.id]).count).to eq(2)
end
it 'does not duplicate an existing primary name in the new language' do
pending '日本語以外の Locale 有効化は別課題で実装する'
prepare_english_reference!
prepare_ainu_reference!
tag = create(:tag, primary_name: 'already_named')
existing = create(:tag_name, name: 'existing_english', tag:,
language_code: 'en', script_code: 'Latn')
existing = create(:tag_name, name: 'existing_ainu', tag:,
language_code: 'ain', script_code: 'Kana')
versions_before = tag.tag_versions.order(:version_no).map(&:attributes)
create_english_locale!
create_ainu_locale!
expect(TagName.where(tag_id: tag.id, language_code: 'en', primary_flg: true))
expect(TagName.where(tag_id: tag.id, language_code: 'ain', primary_flg: true))
.to contain_exactly(existing)
expect(existing.reload).to have_attributes(
tag_id: tag.id, language_code: 'en', script_code: 'Latn',
tag_id: tag.id, language_code: 'ain', script_code: 'Kana',
primary_flg: true, auto_generated: false)
expect(tag.tag_versions.order(:version_no).map(&:attributes)).to eq(versions_before)
end
it 'creates a primary name when the language has only an alias' do
pending '日本語以外の Locale 有効化は別課題で実装する'
prepare_english_reference!
tag = create(:tag, primary_name: 'alias_only_tag')
alias_name = create(:tag_name, :alias, name: 'alias_en',
tag:,
language_code: 'en', script_code: 'Latn')
it 'adds the new language primary while preserving existing names and aliases' do
tag = create(:tag, primary_name: 'named_tag')
create(:tag_name, :alias, name: 'existing_alias', tag:,
language_code: 'ja', script_code: 'Jpan')
original_names = tag.tag_names.order(:id).map(&:attributes)
create_english_locale!
create_ainu_locale!
expect(alias_name.reload).to have_attributes(
tag_id: tag.id,
language_code: 'en', script_code: 'Latn', primary_flg: false)
generated = TagName.find_by!(tag_id: tag.id, language_code: 'en', primary_flg: true)
expect(generated).to have_attributes(tag_id: tag.id, language_code: 'en',
script_code: 'Latn', primary_flg: true,
expect(tag.tag_names.where(language_code: 'ja').order(:id).map(&:attributes))
.to eq(original_names)
generated = TagName.find_by!(tag_id: tag.id, language_code: 'ain', primary_flg: true)
expect(generated).to have_attributes(name: 'named_tag', tag_id: tag.id, language_code: 'ain',
script_code: 'Kana', primary_flg: true,
auto_generated: true)
expect(TagName.where(tag_id: tag.id, language_code: 'en', primary_flg: true).count).to eq(1)
expect(TagName.where(tag_id: tag.id, language_code: 'ain', primary_flg: true).count).to eq(1)
end
it 'rolls back the locale and all generated names when backfill fails' do
pending '日本語以外の Locale 有効化は別課題で実装する'
prepare_english_reference!
prepare_ainu_reference!
first = create(:tag, primary_name: 'atomic_first')
second = create(:tag, primary_name: 'atomic_second')
versions_before = TagVersion.order(:id).map(&:attributes)
allow(TagName).to receive(:generate_name).and_call_original
allow(TagName).to receive(:generate_name)
.with(kind_of(described_class), second, primary_tag_name_for(second, 'ja'))
.and_raise('backfill failed')
expect { create_english_locale! }.to raise_error('backfill failed')
expect { create_ainu_locale! }.to raise_error('backfill failed')
expect(described_class.exists?(code: 'en')).to be(false)
expect(TagName.where(language_code: 'en', tag_id: [first.id, second.id])).to be_empty
expect(described_class.exists?(code: 'ain-Kana')).to be(false)
expect(TagName.where(language_code: 'ain', tag_id: [first.id, second.id])).to be_empty
expect(TagVersion.order(:id).map(&:attributes)).to eq(versions_before)
end
it 'accepts a stable locale code outside the historical enum values' do
pending '日本語以外の Locale 有効化は別課題で実装する'
prepare_english_reference!
it 'records a separate automatic update for each tag without changing its authored create' do
author = create(:user, :admin)
tags = ['locale_history_first', 'locale_history_second'].map do |name|
Tag.find_or_create_by_tag_name!(Locale.nipponese, name, category: :general,
created_by_user: author)
end
originals = tags.map { |tag| tag.tag_versions.sole.attributes }
locale = described_class.create!(
code: 'en-x-rspec', language_code: 'en',
script_code: 'Latn', region_code: 826,
variants: 'x-rspec', name: 'English')
expect { create_ainu_locale! }.to change(TagVersion, :count).by(2)
expect(locale.reload).to have_attributes(
code: 'en-x-rspec', language_code: 'en', script_code: 'Latn',
region_code: 826, variants: 'x-rspec')
tags.zip(originals).each do |tag, original|
versions = tag.tag_versions.order(:version_no).to_a
expect(versions.map(&:event_type)).to eq(['create', 'update'])
expect(versions.first.attributes).to eq(original)
expect(versions.first.created_by_user_id).to eq(author.id)
expect(versions.last).to have_attributes(version_no: 2, created_by_user_id: nil)
expect(versions.last.names.fetch('ain')).to eq(
'name' => tag.name('ja'), 'script_code' => 'Kana',
'auto_generated' => true, 'aliases' => [])
expect(versions.first.names.keys).to eq(['ja'])
expect(tag.reload.version_no).to eq(2)
end
expect { TagNameCompletion.for_locale!(Locale.find('ain-Kana')) }
.not_to change(TagVersion, :count)
end
it 'registers the future British English locale tuple' do
pending '日本語以外の Locale 有効化は別課題で実装する'
it 'records automatic before and after snapshots for a tag without any history' do
tag = create(:tag, primary_name: 'legacy_before_locale')
expect(tag.tag_versions).to be_empty
expect(locale).to have_attributes(code: 'en', language_code: 'en',
script_code: 'Latn', region_code: 826,
create_ainu_locale!
versions = tag.tag_versions.order(:version_no).to_a
expect(versions.map(&:event_type)).to eq(['create', 'update'])
expect(versions.map(&:created_by_user_id)).to eq([nil, nil])
expect(versions.first.names.keys).to eq(['ja'])
expect(versions.last.names.keys).to match_array(['ja', 'ain'])
expect(versions.last.names.fetch('ja')).to eq(versions.first.names.fetch('ja'))
end
it 'rolls back earlier tag names and histories when a later history fails' do
first = create(:tag, primary_name: 'history_atomic_first')
second = create(:tag, primary_name: 'history_atomic_second')
names_before = TagName.order(:id).map(&:attributes)
versions_before = TagVersion.order(:id).map(&:attributes)
allow(TagVersioning).to receive(:record!).and_wrap_original do |original, tag, **options|
if tag.id == second.id && options[:event_type] == :update
raise ActiveRecord::RecordNotSaved, 'history failed'
end
original.call(tag, **options)
end
expect { create_ainu_locale! }
.to raise_error(ActiveRecord::RecordNotSaved, 'history failed')
expect(described_class.exists?(code: 'ain-Kana')).to be(false)
expect(TagName.order(:id).map(&:attributes)).to eq(names_before)
expect(TagVersion.order(:id).map(&:attributes)).to eq(versions_before)
expect([first.reload.version_no, second.reload.version_no]).to eq([1, 1])
end
it 'rejects a private locale code outside the supported locale set' do
prepare_ainu_reference!
expect {
described_class.create!(
code: 'ain-Kana-x-rspec', language_code: 'ain',
script_code: 'Kana', region_code: 392,
variants: 'x-rspec', name: 'Ainu')
}.to raise_error(ArgumentError)
end
it 'registers the supported Ainu locale tuple' do
expect(locale).to have_attributes(code: 'ain-Kana', language_code: 'ain',
script_code: 'Kana', region_code: 392,
variants: '')
end
end

変更されたファイルが多すぎるため、一部のファイルは表示されません さらに表示