arsvendg/Stirling-PDF
1
0
Fork 0
mirror of https://github.com/arsvendg/Stirling-PDF.git synced 2026-07-16 19:33:11 +02:00
Code Issues Actions 4 Packages Projects Releases 1 Wiki Activity

V1 merge (#5193)

Browse Source 
# Description of Changes

<!--
Please provide a summary of the changes, including:

- What was changed
- Why the change was made
- Any challenges encountered

Closes #(issue_number)
-->

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/DeveloperGuide.md#6-testing)
for more details.

---------

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Balázs Szücs <[email protected]>
Signed-off-by: stirlingbot[bot] <stirlingbot[bot]@users.noreply.github.com>
Co-authored-by: ConnorYoh <[email protected]>
Co-authored-by: Connor Yoh <[email protected]>
Co-authored-by: OUNZAR Aymane <[email protected]>
Co-authored-by: YAOU Reda <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: stirlingbot[bot] <195170888+stirlingbot[bot]@users.noreply.github.com>
Co-authored-by: Balázs Szücs <[email protected]>
Co-authored-by: Ludy <[email protected]>
Co-authored-by: tkymmm <[email protected]>
Co-authored-by: Peter Dave Hello <[email protected]>
Co-authored-by: albanobattistella <[email protected]>
Co-authored-by: PingLin8888 <[email protected]>
Co-authored-by: FdaSilvaYY <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: OteJlo <[email protected]>
Co-authored-by: Angel <[email protected]>
Co-authored-by: Ricardo Catarino <[email protected]>
Co-authored-by: Luis Antonio Argüelles González <[email protected]>
Co-authored-by: Dawid Urbański <[email protected]>
Co-authored-by: Stephan Paternotte <[email protected]>
Co-authored-by: Leonardo Santos Paulucio <[email protected]>
Co-authored-by: hamza khalem <[email protected]>
Co-authored-by: IT Creativity + Art Team <[email protected]>
Co-authored-by: Reece Browne <[email protected]>
Co-authored-by: James Brunton <[email protected]>
Co-authored-by: Victor Villarreal <[email protected]>
This commit is contained in:
Anthony Stirling
2025-12-21 10:40:32 +00:00
committed by GitHub
co-authored by ConnorYoh Connor Yoh OUNZAR Aymane YAOU Reda dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> stirlingbot[bot] <195170888+stirlingbot[bot]@users.noreply.github.com> Balázs Szücs Ludy tkymmm Peter Dave Hello albanobattistella PingLin8888 FdaSilvaYY Copilot OteJlo Angel Ricardo Catarino Luis Antonio Argüelles González Dawid Urbański Stephan Paternotte Leonardo Santos Paulucio hamza khalem IT Creativity + Art Team Reece Browne James Brunton Victor Villarreal
parent a5dcdd5bd9
commit 68ed54e398
343 changed files with 25208 additions and 6588 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/counter_translation_v3.py
+297 -88
View File
@@ -1,21 +1,56 @@
"""A script to update language progress status in README.md based on
TOML translation file comparison.
"""
A script to update language progress status in README.md based on
properties file comparison.
This script compares the default translation TOML file with others in the locales directory to
determine language progress.
It then updates README.md based on provided progress list.
This script compares the default (reference) properties file, usually
`messages_en_GB.properties`, with other translation files in the
`app/core/src/main/resources/` directory.
It determines how many lines are fully translated and automatically updates
progress badges in the `README.md`.
Additionally, it maintains a TOML configuration file
(`scripts/ignore_translation.toml`) that defines which keys are ignored
during comparison (e.g., static values like `language.direction`).
Author: Ludy87
Updated for TOML format
Example:
To use this script, simply run it from command line:
$ python counter_translation_v3.py
""" # noqa: D205
Usage:
Run this script directly from the project root.
# --- Compare all translation files and update README.md ---
$ python scripts/counter_translation.py
This will:
• Compare all files matching messages_*.properties
• Update progress badges in README.md
• Update/format ignore_translation.toml automatically
# --- Check a single language file ---
$ python scripts/counter_translation.py --lang messages_fr_FR.properties
This will:
• Compare the French translation file against the English reference
• Print the translation percentage in the console
# --- Print ONLY the percentage (for CI pipelines or automation) ---
$ python scripts/counter_translation.py --lang messages_fr_FR.properties --show-percentage
Example output:
87
Arguments:
-l, --lang <file> Specific properties file to check
(relative or absolute path).
--show-percentage Print only the percentage (no formatting, ideal for CI/CD).
--show-missing-keys Show the list of missing keys when checking a single language file.
"""
import argparse
import glob
import os
import re
import sys
from typing import Iterable
import tomlkit
import tomlkit.toml_file
@@ -23,14 +58,15 @@ import tomlkit.toml_file
def convert_to_multiline(data: tomlkit.TOMLDocument) -> tomlkit.TOMLDocument:
"""Converts 'ignore' and 'missing' arrays to multiline arrays and sorts the first-level keys of the TOML document.
Enhances readability and consistency in the TOML file by ensuring arrays contain unique and sorted entries.
Parameters:
Args:
data (tomlkit.TOMLDocument): The original TOML document containing the data.
Returns:
tomlkit.TOMLDocument: A new TOML document with sorted keys and properly formatted arrays.
""" # noqa: D205
"""
sorted_data = tomlkit.document()
for key in sorted(data.keys()):
value = data[key]
@@ -53,16 +89,19 @@ def convert_to_multiline(data: tomlkit.TOMLDocument) -> tomlkit.TOMLDocument:
def write_readme(progress_list: list[tuple[str, int]]) -> None:
"""Updates the progress status in the README.md file based
on the provided progress list.
"""Updates the progress status in the README.md file based on the provided progress list.
Parameters:
This function reads the existing README.md content, identifies lines containing
language-specific progress badges, and replaces the percentage values and URLs
with the new progress data.
Args:
progress_list (list[tuple[str, int]]): A list of tuples containing
language and progress percentage.
language codes (e.g., 'fr_FR') and progress percentages (integers from 0 to 100).
Returns:
None
""" # noqa: D205
"""
with open("README.md", encoding="utf-8") as file:
content = file.readlines()
@@ -80,70 +119,111 @@ def write_readme(progress_list: list[tuple[str, int]]) -> None:
file.writelines(content)
def parse_toml_file(file_path):
"""
Parses a TOML translation file and returns a flat dictionary of all keys.
:param file_path: Path to the TOML file.
:return: Dictionary with flattened keys and values.
"""
with open(file_path, "r", encoding="utf-8") as file:
data = tomlkit.parse(file.read())
def load_reference_keys(default_file_path: str) -> set[str]:
"""Reads all keys from the reference properties file (excluding comments and empty lines).
def flatten_dict(d, parent_key="", sep="."):
items = {}
for k, v in d.items():
new_key = f"{parent_key}{sep}{k}" if parent_key else k
if isinstance(v, dict):
items.update(flatten_dict(v, new_key, sep=sep))
else:
items[new_key] = v
return items
This function skips the first 5 lines (assumed to be headers or metadata) and then
extracts keys from lines containing '=' separators, ignoring comments (#) and empty lines.
It also handles potential BOM (Byte Order Mark) characters.
return flatten_dict(data)
Args:
default_file_path (str): The path to the default (reference) properties file.
Returns:
set[str]: A set of unique keys found in the reference file.
"""
keys: set[str] = set()
with open(default_file_path, encoding="utf-8") as f:
# Skip the first 5 lines (headers)
for _ in range(5):
try:
next(f)
except StopIteration:
break
for line in f:
s = line.strip()
if not s or s.startswith("#") or "=" not in s:
continue
k, _ = s.split("=", 1)
keys.add(k.strip().replace("\ufeff", "")) # BOM protection
return keys
def _lang_from_path(file_path: str) -> str:
"""Extracts the language code from a properties file path.
Assumes the filename format is 'messages_<language>.properties', where <language>
is the code like 'fr_FR'.
Args:
file_path (str): The full path to the properties file.
Returns:
str: The extracted language code.
"""
return (
os.path.basename(file_path).split("messages_", 1)[1].split(".properties", 1)[0]
)
def compare_files(
default_file_path, file_paths, ignore_translation_file
default_file_path: str,
file_paths: Iterable[str],
ignore_translation_file: str,
show_missing_keys: bool = False,
show_percentage: bool = False,
) -> list[tuple[str, int]]:
"""Compares the default TOML translation file with other
translation files in the locales directory.
"""Compares the default properties file with other properties files in the directory.
Parameters:
default_file_path (str): The path to the default translation TOML file.
file_paths (list): List of paths to translation TOML files.
ignore_translation_file (str): Path to the TOML file with ignore rules.
This function calculates translation progress for each language file by comparing
keys and values line-by-line, skipping headers. It accounts for ignored keys defined
in a TOML configuration file and updates that file with cleaned ignore lists.
English variants (en_GB, en_US) are hardcoded to 100% progress.
Args:
default_file_path (str): The path to the default properties file (reference).
file_paths (Iterable[str]): Iterable of paths to properties files to compare.
ignore_translation_file (str): Path to the TOML file with ignore/missing configurations per language.
show_missing_keys (bool, optional): If True, prints the list of missing keys for each file. Defaults to False.
show_percentage (bool, optional): If True, suppresses detailed output and focuses on percentage calculation. Defaults to False.
Returns:
list[tuple[str, int]]: A list of tuples containing
language and progress percentage.
""" # noqa: D205
default_keys = parse_toml_file(default_file_path)
num_keys = len(default_keys)
list[tuple[str, int]]: A sorted list of tuples containing language codes and progress percentages
(descending order by percentage). Duplicates are removed.
"""
# Count total translatable lines in reference (excluding empty and comments)
num_lines = sum(
1
for line in open(default_file_path, encoding="utf-8")
if line.strip() and not line.strip().startswith("#")
)
result_list = []
ref_keys: set[str] = load_reference_keys(default_file_path)
result_list: list[tuple[str, int]] = []
sort_ignore_translation: tomlkit.TOMLDocument
# read toml
with open(ignore_translation_file, encoding="utf-8") as f:
sort_ignore_translation = tomlkit.parse(f.read())
# Read or initialize TOML config
if os.path.exists(ignore_translation_file):
with open(ignore_translation_file, encoding="utf-8") as f:
sort_ignore_translation = tomlkit.parse(f.read())
else:
sort_ignore_translation = tomlkit.document()
for file_path in file_paths:
# Extract language code from directory name
locale_dir = os.path.basename(os.path.dirname(file_path))
language = _lang_from_path(file_path)
# Convert locale format from hyphen to underscore for TOML compatibility
# e.g., en-GB -> en_GB, sr-LATN-RS -> sr_LATN_RS
language = locale_dir.replace("-", "_")
fails = 0
if language in ["en_GB", "en_US"]:
result_list.append(("en_GB", 100))
result_list.append(("en_US", 100))
# Hardcode English variants to 100%
if "en_GB" in language or "en_US" in language:
result_list.append((language, 100))
continue
# Initialize language table in TOML if missing
if language not in sort_ignore_translation:
sort_ignore_translation[language] = tomlkit.table()
# Ensure default ignore list if empty
if (
"ignore" not in sort_ignore_translation[language]
or len(sort_ignore_translation[language].get("ignore", [])) < 1
@@ -152,53 +232,182 @@ def compare_files(
["language.direction"]
)
current_keys = parse_toml_file(file_path)
# Clean up ignore list to only include keys present in reference
sort_ignore_translation[language]["ignore"] = [
key
for key in sort_ignore_translation[language]["ignore"]
if key in ref_keys or key == "language.direction"
]
# Compare keys
for default_key, default_value in default_keys.items():
if default_key not in current_keys:
# Key is missing entirely
if default_key not in sort_ignore_translation[language]["ignore"]:
print(f"{language}: Key '{default_key}' is missing.")
fails += 1
elif (
default_value == current_keys[default_key]
and default_key not in sort_ignore_translation[language]["ignore"]
fails = 0
missing_str_keys: list[str] = []
with (
open(default_file_path, encoding="utf-8") as default_file,
open(file_path, encoding="utf-8") as file,
):
# Skip headers (first 5 lines) in both files
for _ in range(5):
next(default_file)
try:
next(file)
except StopIteration:
fails = num_lines
break
for line_num, (line_default, line_file) in enumerate(
zip(default_file, file), start=6
):
# Key exists but value is untranslated (same as reference)
print(f"{language}: Key '{default_key}' is missing the translation.")
fails += 1
elif default_value != current_keys[default_key]:
# Key is translated, remove from ignore list if present
if default_key in sort_ignore_translation[language]["ignore"]:
sort_ignore_translation[language]["ignore"].remove(default_key)
try:
# Ignoring empty lines and lines starting with #
if line_default.strip() == "" or line_default.startswith("#"):
continue
default_key, default_value = line_default.split("=", 1)
file_key, file_value = line_file.split("=", 1)
default_key = default_key.strip()
default_value = default_value.strip()
file_key = file_key.strip()
file_value = file_value.strip()
if (
default_value == file_value
and default_key
not in sort_ignore_translation[language]["ignore"]
):
# Missing translation (same as default and not ignored)
fails += 1
missing_str_keys.append(default_key)
if default_value != file_value:
if default_key in sort_ignore_translation[language]["ignore"]:
# Remove from ignore if actually translated
sort_ignore_translation[language]["ignore"].remove(
default_key
)
except ValueError as e:
print(f"Error processing line {line_num} in {file_path}: {e}")
print(f"{line_default}|{line_file}")
sys.exit(1)
except IndexError:
# Handle mismatched line counts
fails += 1
continue
if show_missing_keys:
if len(missing_str_keys) > 0:
print(f" Missing keys: {missing_str_keys}")
else:
print(" No missing keys!")
if not show_percentage:
print(f"{language}: {fails} out of {num_lines} lines are not translated.")
print(f"{language}: {fails} out of {num_keys} keys are not translated.")
result_list.append(
(
language,
int((num_keys - fails) * 100 / num_keys),
int((num_lines - fails) * 100 / num_lines),
)
)
# Write cleaned and formatted TOML back
ignore_translation = convert_to_multiline(sort_ignore_translation)
with open(ignore_translation_file, "w", encoding="utf-8", newline="\n") as file:
file.write(tomlkit.dumps(ignore_translation))
# Remove duplicates and sort by percentage descending
unique_data = list(set(result_list))
unique_data.sort(key=lambda x: x[1], reverse=True)
return unique_data
if __name__ == "__main__":
directory = os.path.join(os.getcwd(), "frontend", "public", "locales")
translation_file_paths = glob.glob(os.path.join(directory, "*", "translation.toml"))
reference_file = os.path.join(directory, "en-GB", "translation.toml")
def main() -> None:
"""Main entry point for the script.
scripts_directory = os.path.join(os.getcwd(), "scripts")
Parses command-line arguments and either processes a single language file
(with optional percentage output) or all files and updates the README.md.
Command-line options:
--lang, -l <file>: Specific properties file to check (e.g., 'messages_fr_FR.properties').
--show-percentage: Print only the translation percentage for --lang and exit.
--show-missing-keys: Show the list of missing keys when checking a single language file.
"""
parser = argparse.ArgumentParser(
description="Compare i18n property files and optionally update README badges."
)
parser.add_argument(
"--lang",
"-l",
help=(
"Specific properties file to check, e.g. 'messages_fr_FR.properties'. "
"If a relative filename is given, it is resolved against the resources directory."
),
)
parser.add_argument(
"--show-percentage",
"-sp",
action="store_true",
help="Print ONLY the translation percentage for --lang and exit.",
)
parser.add_argument(
"--show-missing-keys",
"-smk",
action="store_true",
help="Show the list of missing keys when checking a single language file.",
)
args = parser.parse_args()
# Project layout assumptions
cwd = os.getcwd()
resources_dir = os.path.join(cwd, "app", "core", "src", "main", "resources")
reference_file = os.path.join(resources_dir, "messages_en_GB.properties")
scripts_directory = os.path.join(cwd, "scripts")
translation_state_file = os.path.join(scripts_directory, "ignore_translation.toml")
write_readme(
compare_files(reference_file, translation_file_paths, translation_state_file)
if args.lang:
# Resolve provided path
lang_input = args.lang
if os.path.isabs(lang_input) or os.path.exists(lang_input):
lang_file = lang_input
else:
lang_file = os.path.join(resources_dir, lang_input)
if not os.path.exists(lang_file):
print(f"ERROR: Could not find language file: {lang_file}")
sys.exit(2)
results = compare_files(
reference_file,
[lang_file],
translation_state_file,
args.show_missing_keys,
args.show_percentage,
)
# Find the exact tuple for the requested language
wanted_key = _lang_from_path(lang_file)
for lang, pct in results:
if lang == wanted_key:
if args.show_percentage:
# Print ONLY the number
print(pct)
return
else:
print(f"{lang}: {pct}% translated")
return
# Fallback (should not happen)
print("ERROR: Language not found in results.")
sys.exit(3)
# Default behavior (no --lang): process all and update README
messages_file_paths = glob.glob(
os.path.join(resources_dir, "messages_*.properties")
)
progress = compare_files(
reference_file, messages_file_paths, translation_state_file
)
write_readme(progress)
if __name__ == "__main__":
main()