Copr blog

First look at the Copr MCP server

Jakub Kadlčík — Thu, 02 Apr 2026 00:00:00 +0000

Fedora RISC-V 64-bit now available in Copr

Copr Team — Wed, 28 Jan 2026 00:00:00 +0000

We are happy to announce that Fedora RISC-V 64-bit (riscv64) build targets are now available in Copr! The following chroots have been added to mock-core-configs and enabled in Copr:

fedora-42-riscv64
fedora-43-riscv64

Important notes

We currently do not have native RISC-V hardware in our infrastructure. All riscv64 builds are performed using QEMU emulation on x86_64 machines. Because of this:

Expect longer build times compared to native architectures
Only userspace calls are possible. If your package needs to make a kernelspace call during the build time or the check phase, then your build will fail. Only a few packages are limited by this.

If you encounter any issues, please let us know!

CPT - look back at 2025

Miroslav Suchý — Mon, 22 Dec 2025 00:00:00 +0000

CPT - look back at 2025

We did five releases of Copr that included these new features:

You can regenerate a token from the CLI
Anubis as protection from heartless AI crawlers
Deprecation of modularity
Pulp as a backend storage
Integration with the Log Detective service
Migration to OIDC authentication.
Custom webhooks add a record to the Webhook history table.

We did eight releases of Mock with these features:

Buildroot lockfile generator to enable hermetic builds.
New plugin unbreq to detect unused BuildRequires.
We passed `–suppress-sync=yes` to `systemd-nspawn` but then we disabled it as it caused problems to some packages. It is available as opt-in.
Automatic mapping to the appropriate user-static QEMU binary when building for new architectures (like RISC-V).
Numerous new configs have been added.

We wrote two articles for Fedora Magazine and six other blogposts.
Pavel developed norpm, which allows safe parsing of SPEC files.
We handled several issues in labs where Copr builders are: insufficient resources for s390x, problems with networking of ppc64le, moving x86_64 builders to a different datacenter, allocated high-performing builders for ppc64le…
Jakub added descriptions to resource pools so we can thank our donors.
Jakub wrote a nice page describing why you should use Mock.
Pavel upstreamed Tekton Pipelines for Konflux.
You built 1.4 million packages in Copr in 2025. And Copr serves about 44TB of DNF repositories.

Released SW

Besides Copr, we maintain several projects and this year we did these notable releases:

Five releases of resalloc.
We spent a lot of time maintaining llama-cpp-python in Fedora.
Three releases of fedora-distro-aliases.
One release of rpmconf.
Nine releases of distribution-gpg-keys.
Three releases of fedora-upgrade.
One release of fedora-review.

Log Detective

We successfully launched Log Detective as a prototype in January. Put it fully in production mode in May. And we have been improving it since then.
Jiri set up metrics to measure whether changes in LLM models are improving responses.

SPDX

Miroslav continued working on SPDX Change:
- With the help of Richard Fontana, Miroslav added 75 new entries to fedora-license-data.
- Miroslav added (with the help of Jilayne Lovejoy) 24 new licenses to SPDX license-list’s upstream repository.

Talks

Pavel spoke at “specfile library + norpm” at Flock.
Jirka Podivin participated in a talk We need a Detective to inspect these Logs :: Flock to Fedora 2025 and in a workshop Log Detective Workshop :: Flock to Fedora 2025
Miroslav talked about “Real dangers of AI” at DevConf.cz.
Jiri Podivin presented LogDetective to a delegation from the European Parliament as one of the development projects in Brno.
Jakub did two presentations for PyLadies.

What can you expect from us in 2026?

We finished the development phase of Pulp integration. The first month of 2026 will be focused on operation - migrating users’ data to the Pulp backend.

We want to put rpmeta in production and dynamically allocate high-performance builders when your build will take too much time on a normal builder.

If you want to affect what else we do in 2026, you can check our backlog and cast your vote (use an emoticon for comment #0) or add a comment on how much time the feature will save if implemented. And we will choose the card that will have the most significant impact.

Highlights from 2025-12-10 release

Copr Team — Wed, 10 Dec 2025 00:00:00 +0000

Migrating Copr results to Pulp

Jakub Kadlčík — Mon, 01 Dec 2025 00:00:00 +0000

Over the last year, the Copr team has put a lot of effort into supporting Pulp as a storage for Copr build results. It is now time to realize the investment.

Status Quo

At this very moment, Copr hosts ~35k projects from ~8k Fedora users. The data consumption sits around ~37TB with only ~3TB of free disk space. With a very few exceptions, everything is stored in our “Copr backend” storage.

Our current storage is a RAID array of block devices with the Ext4 filesystem. For various reasons, AWS block devices larger than 16TB are either hard to maintain or expensive. We plan to solve this problem by migrating to an S3 object storage and Pulp, which is a mature and durable-by-default solution for managing software repositories.

We have already migrated all Copr team projects to Pulp to dogfood the integration on ourselves, and we have already migrated one large project to avoid running out of disk space.

The Goal

Our goal is to stop using the current “Copr backend” storage for all new projects and migrate all existing project results to Pulp.

Impact On Users

No change to your workflows should be needed.
Copr repositories enabled on user machines should continue working.
While your projects are being migrated, all your builds and actions will be stuck in the “pending” state until the migration is finished.
After the migration of your projects is finished, you shouldn’t notice any difference from the status quo.
For technical reasons, we won’t notify users directly that their projects are about to be migrated. Please see the Pulp Migration Schedule to get a rough estimate of when you could be affected.
Every user should be affected for at most 24 hours.

To summarize, the change should be transparent to you, there are no UI/UX changes, and your builds may be paused while your project is being migrated.

Our Plan

For a detailed, step-by-step plan with dates and tracked progress, please see the Pulp Migration Schedule. Roughly, the plan is as follows:

Set the default storage for all new Packit projects to Pulp.
Set the default storage for all new projects to Pulp.
Migrate large projects (17 projects that are 100GB+).
Migrate all projects with a few exceptions.
- Migration will be done in alphabetical order by owner name.
- Exceptions: Packit projects, @rubygems/rubygems, @copr/PyPI, and @copr/PyPI3.
Migrate everything that remains.

Contact

If you have any questions or worries regarding our migration of data to Pulp, please let us know:

Generating dataset from Fedora packaging guidelines

Jiri Podivin — Mon, 06 Oct 2025 00:00:00 +0000

Recently we have published a dataset of sourced questions and answers about Fedora packaging guidelines to Hugging Face. With the intention of using it to fine tune LLMs.

In this blog post, I will describe the approach I have taken to generate it. Focusing on matters of prompt construction, and response constraints.

Question of data

Fine tuning large language models to behave according to user instruction, requires a very specific structure of training data used. One can not simply take various documents, no matter how well formed or sourced, and use them as targets for training algorithm.

Doing so, would simply teach the model to reproduce the very same documents, in the best case. Far more probably, the model would produce many variations of the documents given during training. Some are more plausible than others. But all invariably useless.

Instead we need to provide model with inputs and outputs in the same format that will be present, and expected, during operation. That is, conversation samples. With questions, as if asked by user, and answers, grounded in source documents.

The question remains, how to accomplish this at scale. Fine tuning, while cheaper than a full training run, does require many thousands of samples for stable results. Creating samples manually, is simply not viable, even under very optimistic assumptions.

Leveraging LLMs

The answer lies in using LLMs. Language manipulation is what they were created for, after all. Having LLM generate enough samples is a simple matter of writing a prompt, setting a loop and going for a cup of coffee, or going to sleep. That depends on how many samples you want and what kind of hardware is available.

But how can we be sure that model will produce samples that are grounded in reality? After all, if the model already possessed the required domain knowledge, what purpose would any fine tuning serve?

There is also a much more straightforward issue of controlling and parsing the output. The model has to be called many times, and each time the output has to be transformed into a training sample. This process has to be as reliable as possible. Every call costs us money, or time, and we don’t have an infinite supply of either.

Grounding the LLM

To ensure that samples produced are grounded in truth, we need to provide a trustworthy source for model to use when generating them. This simplifies the task considerably, as the model is now merely transforming one kind of text into another.

With a well structured prompt, we can put strong emphasis on the source, lowering the chance of the model producing ungrounded samples considerably.

System prompt bears the most weight here, priming the model to provide responses we need. Emphasis has to be placed on source of knowledge to be used, and on contents of the output.

At the same time, restrictions are placed on what must not be included in model responses. Otherwise we may end up with samples that are seemingly reasonable, but providing no domain knowledge.

Providing an example is another way to improve the quality of the output.

EXTRACTION_SYSTEM_PROMPT = """
You are a meticulous AI assistant designed for high-precision information extraction.
Your primary function is to create specific, fact-based question-and-answer pairs directly from a provided document.

Your task is to adhere to the following strict principles:

1.  **Absolute Grounding:** Every answer you provide MUST be a direct, verbatim quote from the source document. Do not summarize, paraphrase, or infer information.
2.  **Question Specificity:** Questions must be specific and target concrete details (e.g., names, numbers, locations, reasons, processes, definitions). Avoid overly broad or ambiguous questions that could have multiple interpretations. Questions should target information that is unique to the document.
3.  **Self-Contained Questions:** A question should be understandable on its own, but the answer must only be verifiable by consulting the provided document.
4.  **No External Knowledge:** You must operate exclusively on the information within the document. Do not use any prior knowledge.
5.  **Exclude Metadata:** Do not generate questions about the document's own properties, such as its author, title, filename, publication date, or revision history. Focus exclusively on the subject matter content within the text.
6.  **Output Format:** Your response will be a single JSON object containing a list of Q&A pairs. Each pair will have three keys: "question", "answer", and "question_topic".

Example of a GOOD question:
- "What specific percentage of the budget was allocated to research and development in the final quarter?"

Example of a BAD (too vague) question:
- "What does the document say about the budget?"
"""

The source document can be used as part of much less complex, user level, prompt.

BASE_EXTRACTION_PROMPT = """

Attempt to create as many high quality question / answer pairs from following document as possible.
Multiple variants of the same question / answer pair are allowed. As long as the meaning remains unchanged

Document:
'''
{document}
'''
"""

These principles are generally applicable to use of LLMs in various scenarios. Although there may be subtle differences in effectiveness of various approaches to prompting between models, the basic idea of stating purpose, setting constraints and providing examples, remains largely unchallenged.

Structuring model output

Structured output provides an easy solution to the second problem. It allows users to specify an exact format which the LLM output must follow. This feature has been supported by OpenAI for some time, and it has proliferated among self-hosted, FOSS inference providers, such as vLLM or llama.cpp.

The format can be specified as a JSON schema. With pydantic, we can create a model, which will provide us with both schema specification, and validation.

While the set of JSON features supported by different providers differs, and is never complete, it is generally sufficient for most purposes.

We can specify type and various constraints on fields, such as length or range of values.

For example, this model sets the structure of a single question/answer pair. This would be one sample in our dataset.

class QnAPair(BaseModel):
    """Single Q/A pair with topic."""

    question: str = Field(description="Question about contents of the document.")
    answer: str = Field(description="Answer to the question")
    question_topic: str = Field(description="Topic of the question.")

We can also request multiple samples at the same time. By asking for a list of objects we have already defined.

class QnAList(BaseModel):
    """List of Q/A pairs."""

    questions_and_answers: List[QnAPair] = Field(
        description="List of questions and answers about given document.", min_length=1
    )
    document_topic: str = Field(description="Topic of the document")

But we need more

It is very unlikely that a single pass through source document would produce all the samples we need. At very least, we are likely to fill the model context at some point. Simply executing the same call repeatedly, could work. But it would be liable to produce multiple samples with almost, or completely, identical contents.

In order to reach a reasonable number of samples, we need a way to consistently generate multiple variants of those we already have. And again, LLMs can be leveraged to give us exactly that.

This task is much easier than the original one. Because it isn’t strictly necessary to get more varied answers. Provided that those we have are of sufficiently high quality.

If we want our fine tuned model to be robust enough for deployment, it needs to be able to respond to a wide variety of questions.

With the new format for model responses, we can ensure that we only get questions. Sparing some tokens.

class QuestionVariant(BaseModel):
    """Variant of base question"""

    question: str = Field(description="Variant of an existing question.")


class QuestionVariantList(BaseModel):
    """List of question variants."""

    question_variants: List[QuestionVariant] = Field(
        description="List of question variants.", min_length=1
    )

The prompt also needs adjustments. But established principles still apply here.

VARIATION_SYSTEM_PROMPT = """
You are an AI assistant specializing in linguistics and question reformulation. Your task is to rephrase a given question in multiple ways while strictly preserving its original intent and meaning.

Follow these rules:
1.  **Preserve Core Intent:** All generated questions must request the exact same piece of information as the original. The answer to all variants should be identical to the answer of the original question.
2.  **Maintain Specificity:** Do not make the questions more general or vague. If the original asks for a percentage, the variants must also ask for a percentage.
3.  **Vary Phrasing and Structure:** Use synonyms, change the sentence structure (e.g., active to passive), and reorder clauses to create natural-sounding alternatives.
4.  **Output Format:** Your response will be a single JSON object containing a list of dictionaries. Each dictionary will have only one key: "question".
"""

VARIATION_USER_PROMPT = """
Here is the original question:
{question}

Please generate {n_variants} distinct variants of this question.
"""

Sourcing the questions

Before we can get started with generating the dataset, we should ensure that all samples we create can be tracked back to source.

for qna in parsed_response.questions_and_answers:
    base_qna.append(
        SourcedQnAPair(
            question=qna.question,
            answer=qna.answer,
            question_topic=qna.question_topic,
            source=file_path,
            document_topic=parsed_response.document_topic,
        )
    )

This will help us with quality control. As we can later track down the exact source document used, and verify that information in the answer is indeed accurate.

Putting it all together

With model handling both knowledge extraction and dataset augmentation, the task of building a large Q/A dataset becomes much more tractable.

All that remains is putting these tasks into loops. First for iterating over all source documents, deriving base questions. Second for deriving new question variants.

Depending on structure of source documents, and context size of the LLM, it may be prudent to split documents in sections. A number of heuristics can be employed to that end, starting with simple splitting by certain number sentences, or by section headings.

In the case of Fedora Packaging guidelines, all documents are nicely formatted .adoc files, of limited size, and we can use them without change.

Created questions and answers, all SourcedQnAPair objects supporting serialization, can be easily converted into a list of dictionaries, and finally into a Hugging Face dataset.

# Dicts to dataset
dataset = []
for e in tqdm.tqdm(final_qna):
    row = e.model_dump()
    dataset.append(row)

random.shuffle(dataset)

dataset = datasets.Dataset.from_list(dataset)

Shuffling is largely optional, and can be left on the consumer of the dataset to be applied before the dataset is split into training and testing subsets.

Apart from publishing dataset on Hugging Face, we can save it locally, in any of the supported formats.

dataset.to_parquet("fedora_qna.parquet")

Limitations

While this approach is certainly powerful and useful, it is not perfect. LLMs, even when grounded with source documents and well crafted prompts, are capable of producing text without foundation in input.

Even when this is not the case. Source documents ought to be checked. To verify that no private or sensitive information is present. Otherwise they may leak into newly created dataset and any models trained using it.

Finally, and this is the most important limitation of all. Even the best dataset can not supplant deficient fine tuning procedures. Nor can it balance issues caused by a model that is just too small for the task.

It is always necessary to consider all aspects of fine tuning and setting expectations accordingly.

Fedora Copr builders - September 2025

Pavel Raiskup — Fri, 05 Sep 2025 00:00:00 +0000

Highlights from 2025-08-28 release

Copr Team — Thu, 28 Aug 2025 00:00:00 +0000

Log Detective dataset has been published

Jiri Podivin — Sat, 05 Jul 2025 00:00:00 +0000

Since the start of Log Detective project, we have been gathering annotations for failed RPM builds on our website www.logdetective.com. Our intention was always to publish the collected data under open license[1] and use it to improve experience of packagers across Fedora ecosystem.

After gathering slightly over 1000 contributions we have published our dataset[2] on the Hugging Face platform in a QnA format, to facilitate its use in fine tuning LLMs.

In this post I’ll outline the steps taken to create the dataset, and ways the dataset can be used.

From annotations to dataset

Before uploading the data, it was necessary to preprocess gathered annotations. Comments on some of the submitted logs were exceedingly short and lacking meaningful information. While other records were corrupted. The full source of script used to transform annotations into dataset is present in the dataset repository on Hugging Face, but here we can go over the most pertinent parts.

Individual .json files containing annotations were converted into rows of pandas DataFrame to simplify queries and manipulation. Each of the rows contained the name and contents of the parsed .json file.

For each of the entries, several utility columns were calculated to facilitate further work.

Sanitizing dataset

Original setup of our annotation interface was too permissive, so some submissions were not suitable for use. As a remedy, several simple rules were applied to remove annotations that were unsuitable for further work:

Descriptions of issues and fixes shorter than 10 characters are not useful.
Submissions without annotated snippets are not useful.
Submissions with total length of snippet annotations shorter than 10 characters are not useful.
Submissions with snippets set to None are not useful.

With the last rule covering annotations that were corrupted.

Pandas library allows us to define filters as simple conditionals. Turning the aforementioned rule list into straightforward code:

almost_empty_fix = dataset["how_to_fix_len"] < min_how_to_fix_len
almost_empty_reason = dataset["fail_reason_len"] < min_fail_reason_len
almost_empty_snippet_annotations = (
    dataset["tot_snippet_annot_len"] < min_total_snippet_annot_len
)
none_snippets = dataset["logs"].apply(none_snippet)

All conditionals can then be used as part of a single formula, to filter out all annotations that satisfy any of them.

sparse_annotation_criteria = (
    almost_empty_snippet_annotations
    | almost_empty_reason
    | almost_empty_fix
    | none_snippets
)

Inversion of the result will give us indices of all annotations that satisfy our minimal requirements.

Splitting the into subsets

The dataset was then split into two constituent parts, training and test, this is important for future use in training of LLMs. The dataset was randomly shuffled before the split, to prevent bias in order of annotations.

The seed was set, for replicability, to 42.

def split_dataset(dataset: pd.DataFrame, seed: int):
    """Splits dataset into training and evaluation subset"""
    split = dataset.shape[0] // 5
    dataset = dataset.sample(frac=1.0, random_state=seed)
    train_dataset, test_dataset = dataset[split:], dataset[:split]

    return train_dataset, test_dataset

Transformation into question/answer pairs

Training of models based on artificial neural networks, Large Language Models or otherwise, involves formulating your data in form of x -> y where x is your expected input and y is your expected output.

Most obvious use of dataset, would be to fine tune an existing instruction tuned model, imbuing it with information about RPM build failures. Instruction tuned models, such as ibm-granite/granite-3.3-8b-instruct [3], are trained with data in a chat form. These often come in a form of answers and questions, or instructions and actions performed.

If we want to appropriately leverage existing capabilities of the model, we must follow the same form, or risk catastrophic forgetting.[4] This means that we need to transform our annotations into pairs of questions and answers. Prompts used by Log Detective [5] can serve as bases for these.

Each of our annotations consists of explanation, proposed resolution and one or more log snippet annotations. We can turn each of the snippet annotations into it’s own question/answer pair, and create one more for the resolution and explanation.

This means that our dataset will, effectively, contain more entries than we have annotated logs.

Uploading

Both training and test subsets are combined into a single object and uploaded to Hugging Face.

dataset = {
    "train": datasets.Dataset.from_list(train_dataset),
    "test": datasets.Dataset.from_list(test_dataset),
}

dataset = datasets.DatasetDict(dataset)

dataset.push_to_hub(repo_id=repo_id, private=True, token=hf_token)

Keeping the upload initially private is a good idea, unless you are absolutely certain about correctness of your data sanitation and formatting.

How to use the dataset

To use the new dataset, you can simply clone the repo using git, or access it using Hugging Face datasets library:

from datasets import load_dataset

ds = load_dataset("fedora-copr/log_detective_qna")

[1] https://cdla.dev/permissive-2-0/

[2] https://huggingface.co/datasets/fedora-copr/log_detective_qna

[3] https://huggingface.co/ibm-granite/granite-3.3-8b-instruct

[4] https://en.wikipedia.org/wiki/Catastrophic_interference

[5] https://github.com/fedora-copr/logdetective/blob/main/logdetective/prompts.yml

Copr builders powered by bootc

Jakub Kadlčík — Tue, 17 Jun 2025 00:00:00 +0000

Log Detective Model Analysis

Tomas Tomecek — Mon, 26 May 2025 00:00:00 +0000

Before we start with the comparison, let’s acknowledge a big milestone in the Log Detective project. Our tool now comments on Gitlab merge requests for failed CentOS Stream builds [1]. Here is an example of such a comment. I picked the first one from the list and in this case, the explanation is just wrong. However, upon examining the “Details”, which displays log lines annotated by the drain3 library, you can identify the root cause - files are not present on the expected paths.

Back to the comparison. We have been using the Mistral 0.2 7B model in Log Detective for quite some time. It was updated last time in December 2023. 1.5 years is a long time in the development of LLMs. Therefore we decided to compare how Mistral is doing against:

Gemma 3 12B
Gemini 2.5 flash
Granite 3.3 8B

I published all the results in this Github repo: TomasTomecek/logdetective-model-battleground

This was my first time doing such an analysis. I collected 5 different build failures and compared the results on the same set of logs from different models. I haven’t done this in one session — it took me more than a week. I was also updating the code and configuration along the way. Unfortunately, it caused some of the results to be degraded. Please take these results with a grain of salt.

We will use a failed build of python-typing-extensions in this blog post. The package build runs a test suite in the %check section and this test suite has failed like this:

FAILED (failures=20, errors=12, skipped=15)

Let’s see how these four models processed this build log. Brace yourself for a lot of text. My commentary follows these results.

Granite

The provided snippets cover various aspects of an RPM build process, including testing failures, assertion errors, build completion, and a build error. Here’s a breakdown of each snippet:

Dictionary Analysis: A Python dictionary named test_dict is defined with three key-value pairs. ‘label’ is associated with a string data type, while ‘x’ and ‘y’ are associated with integer data types. There are no errors or additional context mentioned.
Test Failure: The test test_regular_extra_items_legacy from the test suite test_typing_extensions.TypedDictTests failed. The failure occurred at line 1844, but no further details about the cause of the failure are provided.
AssertionError: The failure in test test_regular_extra_items_legacy resulted in an AssertionError at line 1846, specifically due to a mismatch between expected and actual frozensets containing ‘log_path’ and ‘log_level’. The actual cause and location of this discrepancy require further investigation.
Multiple Test Failures: The build process encountered 20 failures, 12 errors, and 15 skipped tests, as indicated by the ‘FAILED’ status with failure statistics at line 1877. This suggests various issues need addressing in the test suite.
Build Check Failure: The build process failed during the ‘%check’ stage at line 1879, with the error message “Bad exit status from /var/tmp/rpm-tmp.5QvbcS (%check)”. This indicates a non-zero exit status from a script or command executed during the build process.
Build Success: The build for ‘python-typing-extensions-4.13.2-1.fc43.src.rpm’ was completed successfully at line 1882, as signified by the build status code (1882,).
Configuration Error: The build failed in the configuration stage for ‘fedora-rawhide-x86_64’ platform at line 1888, with the error message ‘ERROR: Exception(…)’. This suggests an unhandled exception occurred during this specific build attempt.
Command Failure: An error occurred when executing the build command for ‘python-typing-extensions’ on Fedora Rawhide for the x86_64 architecture at line 1894. The command failed, and the build environment setup details are provided for context.
Copr Build Error: The copr build for the package failed without providing further information at line 1897. This indicates that the build process hosted on Copr encountered an issue, resulting in a failure.

In summary, these snippets highlight multiple issues within the RPM build process, including test failures, assertion errors, build check failures, configuration errors, and a copr build error. Each snippet requires further investigation to pinpoint the root causes and develop appropriate solutions.

Mistral

The provided RPM build log snippets indicate several errors and failures during the build process of the python-typing-extensions package on the Fedora rawhide platform. Here’s a summary of the issues:

Testing phase failures and errors:
- Multiple tests (both failures and errors) were encountered during the testing phase, making the testing process unsuccessful.
- The specific tests that failed and the reasons for their failure are not provided in the log snippets.
Packaging issues:
- The ‘rpmbuild’ command execution inside the container environment using systemd-nspawn failed, causing the build to fail. No information about the nature of the error is provided in the log snippets.
Copr build error:
- The build process in the Copr repository also failed, but no error message is given in the provided log snippets.

To diagnose the issue, it is recommended to inspect the system logs and the RPM build output for any error messages that may have been generated during the build process. It’s also important to check if there are any updates or changes to the dependencies and the python-typing-extensions package itself that could cause these issues. Additionally, trying to build the package on a different platform or using a different build environment might help identify if the issue is specific to the Fedora rawhide platform or the systemd-nspawn containerization.

Gemma

The build of the python-typing-extensions package failed due to a series of test failures within the RPM build process. The core issue isn’t immediately clear from a single error message, but the numerous AssertionError exceptions across multiple test suites (test_stock_annotations, test_extra_items_class_arg, test_implicit_any_inheritance, test_regular_extra_items_legacy, test_total) strongly suggest a problem with the package’s type hinting and/or the interaction between different typed dict features. The specific assertion failures indicate discrepancies in expected versus actual set contents, pointing to potential logic errors in how type annotations are being handled or inherited. The final error points to a failing %check script within the RPM build, which is triggered by these test failures.

The root cause seems to lie within the typing_extensions codebase itself, rather than the build environment, although the complex setup of the mock chroot environment might expose these issues. The failure is likely related to a change in how type hints are processed or handled in the typing_extensions library, which is then exposed by the tests.

Recommended Solution:

The most practical solution is to investigate the failing tests in the typing_extensions source code (src/test_typing_extensions.py) and identify the underlying logic errors that are causing the AssertionErrors. Specifically, focus on the tests mentioned in the logs (test_stock_annotations, test_extra_items_class_arg, etc.) to understand the expected behavior versus the actual behavior. Debugging the code, potentially with a local build and debugging environment, would be necessary to pinpoint the cause. A fix might involve adjusting the expected values in the assertions or modifying the code under test to align with the intended behavior. After the underlying code is fixed, a new build should be triggered to confirm the fix resolves the test failures.

Gemini

Based on the log snippets, a failure occurred during the package build process. The build failed specifically during the %check stage, which is where the package’s test suite is run. Multiple tests failed (indicated by FAIL: and AssertionError entries), resulting in the test suite reporting FAILED (failures=20, errors=12, skipped=15). This test failure caused the %check stage script to exit with a bad status, leading to the RPM build errors: error: Bad exit status and the ultimate Copr build error: Build failed.

The recommended solution is to investigate the specific test failures reported in the log (e.g., test_stock_annotations, test_implicit_any_inheritance, test_regular_extra_items_legacy, test_total) and fix the underlying code or test logic issues that are causing them to fail. Once the tests pass successfully, the build should complete without the %check stage failure.

Commentary

As you can see, the results are different. One feedback we are already hearing from engineers is that shorter is better. LLMs can be pretty verbose, so even when we instruct the model to be concise, quite often it’s not.

We’ll start with Gemini. The comment from this model is just perfect. Short, straight to the point, no extra sentences. If I asked a senior engineer for help, this is the message I would want to see. Gemma is also pretty good though it lacks confidence, especially when you compare it to Gemini. They are similar but Gemma produced more words, lacks confidence, and clearly doesn’t understand the structure of the RPM build process.

Let’s continue with Mistral. It reads pretty poor when compared to Gemma, and especially Gemini. It completely lacks details (names of tests), contains irrelevant information, and therefore provides very little value to an engineer. I would absolutely need to see the logs to understand what went wrong.

Granite [2] results are interesting. They read as a summary of the log, which makes sense because that’s how we structure the final prompt. I can see value in this type of output, but at the same time it’s not the correct answer - telling me why the build has failed in a few clear sentences.

Based on this analysis it’s clear to me that we need to upgrade our model. Soon!

[1] This applies to RHEL as well since the builds are 99% identical, hence we decided to pick CS builds because they are completely open
[2] I also added results from Granite 3.1 in the Github repo, though unfortunately they are very poor compared to all the other models

Highlights from 2025-03-26 release

Copr Team — Wed, 26 Mar 2025 00:00:00 +0000

4 cool new projects to try in COPR for March 2025

Jiří Kyjovský — Wed, 05 Mar 2025 00:00:00 +0000

Log Detective can now explain build log failures

Tomas Tomecek — Wed, 15 Jan 2025 00:00:00 +0000

We’re thrilled to announce the new functionality of Log Detective. The service can now explain in detail a build log of your choice. Click on logdetective.com/explain, paste a link to your log, wait one minute and you should see a description of important lines with the overall explanation.

Disclaimer

This is our initial prototype of this functionality. We appreciate all your feedback. Please be patient, as there will be service disruptions, instability, and usability issues.

1️⃣ The service can process only one request at a time. We understand this is very limiting, especially when multiple people will use the service at the same time. You may need to wait several minutes to get a reply in such a case. We are thankful to the Fedora Project for sponsoring our infrastructure needs. Please, reach out to us if you have experience in running LLM inference in parallel on a single GPU. We will scale the service later based on the demand in usage.
🌍 We are using a general-purpose LLM in the background. The data we are collecting are not being utilized just yet in the answers. We still don’t have enough data to fine-tune a model that would provide high-quality answers. Our goal for 2025 is to research and train such a model.
🔬 The functionality is highly experimental. Our team has worked diligently to deliver this prototype. However, this represents only a fraction of the service’s full potential. We eagerly welcome all feedback and would greatly appreciate it if you could report any unusual behavior, errors, or tracebacks that you encounter.

Step-by-step guide

The new functionality is available under “Explain logs”.

https://logdetective.com/explain

The service only accepts links to raw logs; all of these are always available for Copr and Koji builds.

Upon submission, Log Detective retrieves the log file and initiates the processing phase. During this process, the service interacts with the underlying LLM through multiple requests. It usually takes one minute to return an answer.

The result is split into two sections:

The left side contains an overall explanation for the whole log.
Log Detective uses Drain to select important lines from the log. You can see descriptions of them on the right side after you expand them by clicking on a line.

In our example here, we can see the build failed because a required library libtree-sitter.so.0.23()(64bit) is not available and cannot be installed. Log Detective attempts to explain this issue, though it fails with the solution: it’s not as easy as just installing the library with dnf. There is definitely room for improvement in the final answer.

This is where you can help. If you are not satisfied with the answer and know a better one, head over to Log Detective website homepage, enter the log link and annotate it for us. Once we fine-tune our own model, the service can provide much better explanations thanks to you.

You can reach us at the Fedora Copr build tools discussion Matrix channel: #buildsys:fedoraproject.org. Please open issues in: github.com/fedora-copr/logdetective-website/issues/new/choose

CPT - look back at 2024

Miroslav Suchý — Wed, 11 Dec 2024 00:00:00 +0000

Let me sum up what the Community Packaging Team (CPT) did during 2024:

Copr

Most of our focus is on Copr build system. These are notable features and improvements we did in Copr:
IBM allowed us to use eighteen s390x machines for Fedora’s Copr. We added it to pool of builders.
Pavel finalized the “high-performance s390x builders” in Fedora’s Copr, which can be enabled as an opt-in. You can use this feature for x86_64, aarch64 and s390x architectures now. As of today several projects use this feature - that includes projects that build kernel, chromium, QT, texlive or AI packages.
At the February, we started working on integrating Pulp into Copr (blogpost) As of today, a project can be configured to use either the old backend or Pulp backend. We plan to dogfood this feature in December and migrate some of our repositories to Pulp. In January 2025 we plan to make it publicly available as opt-in for early testers (with a documented set of features that still does not work). Here is an example of a project that stores results in Pulp: https://copr.fedorainfracloud.org/coprs/g/copr/pulp-test/ You can try it live.
Jakub updated the Copr screenshot tutorial
Jirka fixed a performance issue caused by a Power9 VM spawning issues in OSUOSL OpenStack
We had another round of discussion with TestingFarm and CKI about sharing the code that manages workers (short lived VMs).
Jakub created a video tutorial documentation page for Copr
We wrote two articles for Fedora Magazine “4 cool new projects to try in Copr”: one in March and second in October
We did four releases of Copr including these features:
- SSH access to Copr builders
- Ability to use your own dedicated builders - no one used this so far.
- Self documenting API using RestX
Jakub wrote a blogpost describing to new contributors how they can start contributing.
Pavel wrote about our new policy for rolling chroots.
Jakub wrote down the plan of how we are going to EOL Modularity in Copr.
Jakub started to investigate an issue with prolonging RPM GPG keys
We worked on lots of issues with Copr’s storage as current Copr storage usage is 33TB and weekly incremental diff is about 4TB.
We worked on [expired GPG keys issue](https://github.com/fedora-copr/copr/issues/2894#issuecomment-2454539349] that resulted in subsequent work in DNF and Fedora Change proposal.
Copr built 1,640,000 builds during the whole year. That is about 3 builds every minute.

Fedora

Some of our side works has notable impact on Fedora Linux:

Pavel and Evan (from DNF team) worked on Fedora Change Build Fedora with DNF 5 - Fedora’s Koji started using DNF5 as base package manager during builds.
Miroslav cleaned 30k AMIs and 30k snapshots from Fedora’s AWS account and drafted next steps in cleanup of old stuff there.
Miroslav tagged all Fedora’s GA images in AWS (in all regions) with proper tag.
Jakub made several improvements to the Fedora Packager Sponsors page.
Miroslav altered his regular emails about Fedora’s cloud spending and resource usage to show dollars spent (believe or not, until now no one had a clue).
Jakub helped the Pulp team to package client tools for Fedora.

SPDX

Miroslav continued working on SPDX Change:
We started the year with 60 % of Fedora packages converted to SPDX (out of 30067 license tags). And we closed the year with 99% tags converted.
Miroslav added 98 new licenses to fedora-license-data (with great help of Richard Fontana).
Miroslav added 70 new licenses to upstream project SPDX.org (with great help of Jilayne Lovejoy).
Miroslav made about 450 PR for Fedora packages correcting the License tags.
Jakub created a proof of concept for continuous license validation In case we need CI for licenses.
Miroslav did package reviews for scancode-toolkit and its dependencies (a lot of them).
Miroslav created script and gitlab action that produces License policy for Scancode Toolkit from Fedora’s data.

Released SW

Beside Copr, we maintain several project and this year we did these notable releases:

Pavel did six releases of Mock and nine releases of Mock core configs. And configured Mock to use TestingFarm for CI. Notable new feature of Mock added this year:
- Hermetic builds
Jakub did three releases of Tito.
Jakub created a new project fedora-distro-aliases and packaged it for Fedora, This is shared code by several projects. Inspired by this, Dan Cermak from SUSE created opensuse-distro-aliases
Pavel did new releases for resalloc (v5.2 and v5.3), resalloc-aws, resalloc-ibm-cloud and resalloc-openstack. Overall stability and convenience fixes, making the “agent spawner” stable.

Logdetective

We launched https://logdetective.com/ website - and started working on AI tool that will tell you why the build fails and give you a hint on how to fix the failing build.
The project lives at https://fedora-copr.github.io/posts/log-detective .
We investigated usage of logjuicer, but then found drain3 as a better tool
In parallel to that we are collecting our own collection that we plan to use as foundation to our own model. Likely using InspectLab.
We packaged a logdetective command line tool for Fedora. Including several AI related dependencies
Jiri Podivin delivered a talk at LinuxDays (in Czech).

Other

Jakub published a blogpost and a video about Vim LSP (Language Server Protocol) created on day of learning.
Jakub added vcs-diff-lint support for Ruff.

What can you expect from us in 2025?

In a few days - maybe still in 2024 - Pavel wants to release a new version of Mock that will have a nice feature Generate and use OCI images for buildroot.

You can check our Kanban board and check the column “in 3 months”. Notably we will work on DNF5 plugin support and design of EPEL10 chroots in Mock.

As noted above we plan to allow as opt-in migration of projects to Pulp backend. This feature has no direct impact on users, but significantly ease our maintenance.

If you want to affect what else we do in 2025 you can check our backlog and cast your vote (use emoticon for comment #0) or add a comment on how much time will save the feature if implemented. And we will choose the card that will have biggest impact.

Copr Modularity, the End of an Era

Jakub Kadlčík — Wed, 23 Oct 2024 00:00:00 +0000

4 cool new projects to try in Copr for October 2024

Jiří Kyjovský — Mon, 21 Oct 2024 00:00:00 +0000

Highlights from 2024-10-03 release

Copr Team — Thu, 03 Oct 2024 00:00:00 +0000

Browse Copr projects from Ulauncher

Jakub Kadlčík — Tue, 10 Sep 2024 00:00:00 +0000

Reviewing annotations in Log Detective

Tomas Tomecek — Fri, 16 Aug 2024 00:00:00 +0000

You may have heard of our Log Detective project. We want to use AI to process your logs of failed builds. The tool tells you where the problem is and how to fix it: you can try this today locally on your laptop using the logdetective CLI tool. In order to provide the best answers in the future, we need to collect annotations of specific failures for what went wrong and how to fix it.

We appreciate everyone who submitted annotations already, thank you! It will allow us to create a model with knowledge about building software, not just RPMs, but also container images, test failures and various language specific buildsystems. I envision that Log Detective will provide useful answers to most of logs with build failures. Though we still have a long way getting there.

So far, the annotations on the logdetective.com site are read-only. Once submitted, there is no way to change them. Everyone can download them, but that’s it.

Starting in August 2024, we have now deployed a review interface on the Log Detective website. This will allow anyone to rate (thumbs up and down) each snippet and therefore curate collected data.

Annotations combined with your rating will enable us create a high-quality data set and increase Log Detective’s chance giving you solid advice.

We’ve already done research on how to utilize this data inside Log Detective but haven’t decided yet. Please stay tuned.

How do I review snippets?

Once you click the “Review logs” button at the top menu, you’ll reach the new review interface. The instructions are on the left side.

You are welcome to add more snippets if you can spot more information that should be annotated. If an annotation is incorrect:

You are welcome to edit the text and provide correct explanation
If the highlight is wrong, please downvote it and create a new annotation.

Flask Restx Migration

Tue, 09 Jul 2024 00:00:00 +0000

Migration from Flask to Flask-RESTx for API documentation

Hey everyone! We recently decided to switch COPR’s API from Flask to Flask-RESTx, mainly to improve our API documentation since the documenting was non-existing.

Why Flask-RESTx?

First off, Flask-RESTx is an extension of Flask that makes building and documenting APIs easier. Here’s why we decided to use it:

It’s wrapper around Flask, so we could re-use much of our code-base for Flask.
It generates a cool, interactive Swagger UI Documentation for COPR API.
Flask-RESTx supports OpenAPI 3.0.
It ensures the data API handles is valid and correctly formatted.

What it looks like

So, what does this all look like in practice? Let me give you a sneak peek.

Instead of static, hard-to-update documentation, you get an interactive interface where users can do these things:

Explore Endpoints: Users can see all available API endpoints at a glance. Each endpoint is listed with a clear description of what it does.
View Details: Clicking on an endpoint reveals detailed information about the parameters it accepts and the responses it returns. This makes it super easy for you to know exactly how to use each endpoint.
Try It Out: One of the coolest features is the ability to test endpoints directly from the documentation. You can fill in parameters and see real-time responses, which helps them understand how everything works without writing a single line of code. Imagine this: you have an endpoint /api_3/builds that returns a list of builds. With Flask-RESTx, you will see this endpoint listed, click on it, and immediately know what parameters they can pass (like page or limit) and what the response will look like (a list of items). Plus, you can test it out right there to see it in action. You can find the docs at copr.fedorainfracloud.org/api_3/docs

This interactive, user-friendly documentation not only makes life easier for your users but also cuts down on the support you need to provide.

Migration Experience

Switching to Flask-RESTx was pretty lenghty process, and here’s what happened:

We decided to migrate the endpoint gradually to test them in bulks in production.
There were some differences between Flask and Flask-RESTx, so there was a lot of compatibility related code to create bridge between those two APIs.
The Swagger documentation is taken from codebase which required a lot of coding to happen.
Flask-RESTx has deprecated parsers, telling us to use something like pydantic or marshmallow to use validation fully. We decided to skip this and write our own parsers.

Switching to Flask-RESTx was a great move for our project. The improved documentation made our API more accessible and user-friendly. If you’re using Flask and want to step up your API documentation game, I highly recommend giving Flask-RESTx a try. The benefits of automatic, interactive documentation and a cleaner codebase are definitely worth it.

Rawhide (aka "rolling") chroot EOL policy in Copr

Pavel Raiskup — Thu, 04 Jul 2024 00:00:00 +0000

In Fedora Copr, we strive to avoid wasting disk storage unnecessarily. Disk storage is the commodity that incurs the highest cloud costs. For example, we clean the build results for EOL chroots, we clean source builds (build results that are imported into DistGit), we clean the DistGit lookaside cache, build logs, uploaded files, failed build logs, and more. Yet, we maintain >= 40TB of data with RAID redundancy, volume snapshots, …

Most of the Fedora Copr backend storage is used for holding the build results related to Fedora, with the largest consumer being the Fedora Rawhide chroot.

The “problem” with Fedora Rawhide as a “rolling” distribution is that it never reaches EOL, unlike branched Fedora releases. For example, take a look at the data consumption for the old Fedora 35 in the previous link if you want to see the difference.

Yet, most existing projects do build for Rawhide – and until now, we did not have a tool for performing any cleanups there. Now we do.

Warning: Fedora Rawhide is not the only target distro suffering from this problem. There are other rolling distros we build for, where we’ll apply the same policy, namely Fedora ELN, openSUSE Tumbleweed, and Mageia Cauldron.

The Rolling Policy Being Implemented

In this Copr issue, we concluded that a viable option is to detect the liveness of particular “rolling” chroots in Copr projects. If a chroot appears lifeless/dead, we will disable it and schedule it for future removal. According to the initial proposed policy, Copr will mark a chroot as “lifeless” after a 6-month period of build inactivity. Such a “lifeless” chroot then enters another 6-month protection period where we keep the built results, but the corresponding project administrators are informed about the future chroot data removal. If the chroot remains inactive during this period, the data will be removed. The cleanup process is extended if the chroot appears active, either by (a) a user building a package in the chroot or (b) a user manually prolonging the chroot validity.

At the time of this announcement, there is no risk of data loss. The corresponding code is not yet deployed in production, and once it is, it will take at least 12 months before we remove any data. We just want to inform you and prepare you to take action.

The default limits (6+6 months) are chosen because Fedora’s default release cycle is about 6 months. During this time, something significant typically happens that makes the built packages somewhat useless or non-installable (e.g., library API changes, changes in generated Provides, anything that makes the package non-installable or non-working). However, we understand that some packages (e.g., data-only packages) could remain working for years without needing a rebuild. If this applies to you, try to estimate how much inconvenience this change brings (the “manual chroot prolonging”), and let us know.

Do you want to contribute to Copr?

Jakub Kadlčík — Thu, 27 Jun 2024 00:00:00 +0000

Need something extra? Employ your own Copr builders!

Pavel Raiskup — Mon, 24 Jun 2024 00:00:00 +0000

Highlights from 2024-05-22 release

Copr Team — Wed, 22 May 2024 00:00:00 +0000

Video tutorial: Using Copr CLI

Jakub Kadlčík — Mon, 13 May 2024 00:00:00 +0000

SSH access to Copr builders

Jakub Kadlčík — Mon, 18 Mar 2024 00:00:00 +0000

RFC: Planning Copr → PULP RPM content movement

Pavel Raiskup — Mon, 18 Mar 2024 00:00:00 +0000

Highlights from 2024-03-07 release

Copr Team — Thu, 07 Mar 2024 00:00:00 +0000

4 cool new projects to try in COPR for March 2024

Miroslav Suchý — Mon, 04 Mar 2024 00:00:00 +0000

AI-driven Build Failure diagnosis

The Log Detective team — Tue, 16 Jan 2024 00:00:00 +0000

We are delighted to introduce Log Detective, an exciting new project at the intersection of artificial intelligence and error pattern detection in build logs. Unlike traditional approaches, Log Detective relies on your valuable input to kickstart its journey.

What is Log Detective?

Log Detective takes a new approach by seeking insights directly from maintainers willing to help other fellow maintainers. Instead of using pre-trained generic models, we want to offer a customized AI system only on sentences and language specific to packaging. Provided by maintainers that are able to identify and understand common error patterns in build logs.

What’s Happening at Log Detective?

Data Collection Period: We want to gather a rich dataset highlighting various error patterns encountered in real-world (RPM and container first) build logs.

Model Training Initiated: With the collected data in hand, we’ll train (or complement/co-train) an error pattern detection model. This pivotal step marks a significant leap towards Log Detective’s goal of providing accurate and efficient solutions. We plan to do this step once we gather more than 1000 samples.

Integration with build-systems: The API of the resulting system/model will allow us later to integrate with Copr and other existing build systems to help you “light-speed” diagnose errors in your future builds.

Re-usability: There will be tooling for using the models e.g. for analyzing Mock build logs locally. This tooling would form a basic building block for the build system integration above, trying to follow the UNIX principles.

User Feedback Loop: Your continued feedback and input will remain crucial. We encourage you to stay engaged, share additional insights, and participate in discussions to refine Log Detective further.

How Your Contributions Shape Log Detective?

We prepared a data-gathering portal. The portal contains a convenient form for you to easily describe the core of a given build failure. The tool is integrated with the Fedora Koji and Fedora Copr build systems, but also with Packit service, so it is sufficient to specify the build/run ID for which you want to provide data and then, using the pre-uploaded build logs, you can just “highlight” the important part of the build log, and provide a human-readable description/reason for the build failure.

Join Us in Shaping the Future!

Visit our GitHub repository, and submit your ideas and comments!

CPT - look back at 2023

Miroslav Suchý — Wed, 13 Dec 2023 00:00:00 +0000

Let me sum up what the Community Packaging Team (CPT) did during 2023:

Copr

We did five releases of Copr and upgraded Copr servers to Fedora 39.
We wrote two “4 cool new projects to try in Copr” articles for Fedora Magazine
We fixed several degradations of Copr performance where users noticed a disruption in our service.
Long builds can be run on high-performance builders now.
Pavel moved s390x from Tokyo to the newly available Washington DC datacenter.
This is cheaper for our sponsor - IBM. And it is closer to other Copr servers.
We ran out of storage in Copr and we had to add 16 TB storage. The storage holding your repositories is now 44 TB big. And it brings us new challenges to maintain it.
Copr helped to discover that RPM Sequoia (F38+) doesn’t accept prolonged Copr GPG signatures. Pavel coordinated the effort to fix it.
We spent lots of time with the Pulp team and identified issues that need to be addressed in Pulp before Copr uses Pulp as a backend. Most of the issues are resolved now and we are looking forward to working on enabling Pulp as a backend next year. The biggest benefits should be:
- Use of S3 as storage - we will get unlimited storage and the maintenance of storage devices will be much easier.
- Delegate performance tunning with createrepo and pruning old data to Pulp project.
openEuler (Huawei) deployed their own instance of Copr and contributed back with several fixes and features. E.g., “group” support for OIDC.
Jirka implemented the repo prioritization feature (DNF will prefer the installation of packages from the Copr project over the distro-default packages)
We moved SRPM introspection from the backend to builders - this enabled us to correctly evaluate various macros and conditions.
Pyp2spec is now the default PyPI spec generator in Copr. Thanks Karolina Surma for her contribution.
Thanks to Jakub, Copr now honors ExclusiveArch and does not report excluded arch as failed.
Jakub created a usage treemap that allowed us to better understand our users.
We made several improvements to the Copr front page to decrease loading time.
Jakub documented Copr-specific RPM macros.
Pavel created the Copr plugin for DNF5.
We documented Pagure integration.
Due to performance issues, we had to disable the generation of appstream metadata by default.
We tested and documented a Copr-backend disaster recovery. We discovered several issues and addressed them.
Jakub implemented ‘copr download-build –review’ that downloads only fedora-review results.
Jiri implemented `copr get PROJECT` that prints information about a specific project.
We stopped producing SQLite repo databases.

Fedora

Pavel got promoted to a ‘sysadmin-main’ role within Fedora Infra.
Pavel participates in the Fedora Change proposal Build Fedora with DNF 5.
Miroslav participated in Fedora Change proposal SPDX Licenses Phase 1 and Phase 2. This included 23 emails with detailed statistics of progress.
Jakub bug fixed modulemd-tools in F39+.
Miroslav cleaned up Fedora’s AWS account and looked for forgotten resources. This required lots of emails and making sure the resources were really leftovers and not used by anything live.
Jakub recorded a video: How to add a package to the official Fedora repositories.
Jakub enhanced FedoraReview to generate machine-readable JSON file.
Jakub started and later enhanced Fedora Review Service.
Jakub kicked off a discussion about autoassing sponsors to newcomers in Fedora.
Jiří added the follow_fedora_branching attribute to project handling to both copr-cli and API.
Due to the constant problems with Pagure, we migrated our git repository to GitHub.
Jakub exposed useful ENV variables to a custom method.
We did a dist-git release.

Mock

Pavel did 13 releases of Mock and mock-core-configs.
Mock now uses container images for bootstrap chroot by default.
Together with the DNF team Pavel worked on DNF5 and checked the compatibility and reported all issues to the DNF team. As a result, you will likely not notice a change when Mock starts to use DNF5 for builds.
Pavel spent a lot of time making sure Mock runs flawlessly in rootles Podman containers.
Mock now pre-creates users in chroot using shadow-utils from a host. Using this, Mock can newly pre-create a configured set of in-chroot user accounts. This made the Pesign support in Mock “supported” and stable.
Nils Philippsen contributed rpmautospec plugin that pre-processes spec files that use rpmautospec features.

SPDX

Miroslav sent 23 emails with detailed statistics of progress.
During this year Fedora contributors migrated 11 347 license tags. Thank you.
Miroslav talked about SPDX licenses at OpenAlt (in Czech).
Miroslav generated BNF grammar from fedora-license-data. This allowed comfortable checks using license-validate without constant rebases.
Miroslav had a talk The case of SPDX vs. Fedora at Flock.
Miroslav had a talk Software Bill of Materials (SBoM) for dummies at DevConf.CZ.
Miroslav did 5 releases of license-validate.
Miroslav did 26 releases of fedora-license-data.
Miroslav submitted 45 pull-requests with new licenses to upstream of SPDX.org.
Miroslav submitted about 120 pull-requests with new licenses to fedora-license-data.

Other

We started working on the AI project Log Detective which will help package maintainers. This has not been announced yet. Stay tuned. Coming soon. Very soon.
Jakub did 5 releases of Tracer.
Pavel did 3 releases of Resalloc
Pavel blogged how to re-partition an almost live system using mdadm and LVM.
Jakub nicely video blogged about how to prepare a patch-set for (CentOS Stream) packages.
Miroslav participated in a workshop Writing blog posts to boost community awareness at DevConf.CZ.
Pavel did a release of argparse-manpage.
Miroslav gave a lightning talk about SPDX ad MiniDevConf.cz. Pavel at the same conference gave a lightning talk about vcs-diff-lint.

Coming soon

Jakub is working on a feature that will allow you to SSH to the Copr builder hosing the build process (e.g. to debug your failing build). Stay tuned!
We are in the process of moving our API to self-documented using flask-restx.

Ideas we have for 2024

Finish issues from our triaged backlog.
Finish SPDX migration (target is October 2024)
Work on Log Detective.
Start using Pulp as a repository backend for Copr.
In Mock, use container images for chroots too. (Right now we use them for bootstrap chroots).
Start Cavil instance - this will allow package license audit after each commit.

The Community Packaging Team consists of Pavel Raiskup, Jakub Kadlčík, Jiří Kyjovský, and me.

Fedora Copr team is moving from IRC to Matrix

Copr Team — Fri, 08 Dec 2023 00:00:00 +0000

Highlights from 2023-08-16 release

Copr Team — Wed, 16 Aug 2023 00:00:00 +0000

4 cool new projects to try in COPR for June 2023

Jakub Kadlčík — Mon, 05 Jun 2023 00:00:00 +0000

Highlights from 2023-05-25 release

Copr Team — Thu, 25 May 2023 00:00:00 +0000

Highlights from 2023-04-05 release

Copr Team — Wed, 05 Apr 2023 00:00:00 +0000

4 cool new projects to try in Copr for March 2023

Jakub Kadlčík — Wed, 01 Mar 2023 00:00:00 +0000

Copr's migration to GitHub

Jiří Kyjovský — Thu, 16 Feb 2023 00:00:00 +0000

For a very long time, Copr mainliners have been considering migrating from Pagure to GitHub. The problem has been the constant problems with Pagure and even though GitHub is not open source, it just works. Not to mention that GitHub is miles ahead of Pagure with plenty of useful features.

So on the migration itself, Copr had a mirror on GitHub at the time, so the code migration wasn’t a problem. The problem was migrating all the issues and pull requests so that their IDs were preserved. We wanted to keep the same IDs for compatibility with the links in the commits. And there is no general solution to migrate all issues and PRs from Pagure to GitHub while preserving the IDs. So that’s why I wrote a temporary script to migrate Copr. Please note that the code was only meant to be used once and only for the Copr migration, so it should not be used by anyone. However, it can serve as an inspiration and I would also like to share with you some of the problems I encountered during the migration.

Choose the right tools

For the whole migration, I used the tool ogr, which provides a unified API for communicating with GitHub, Gitlab, and Pagure. For communication with the GitHub API, you need to set up a GitHub token, and likewise for Pagure. Note the permissions that you give the token, only give it as much as you really need (for example, you probably don’t want to allow it to delete the repository).

And then you just have to start using the tokens

gh_service = GithubService(token=GITHUB_TOKEN)
pagure_service = PagureService(token=PAGURE_TOKEN, instance_url="https://pagure.io")

gh_copr_project = gh_service.get_project(namespace="fedora-copr", repo="copr")
pagure_copr_project = pagure_service.get_project(namespace="copr", repo="copr", username="pagure_nick")

What to migrate

First of all, it is important to determine what we want to migrate. In our case, we wanted to migrate as quickly as possible, so we just needed to migrate all issues and PRs to keep the order of their IDs. To save time, decided to migrate the PRs as issues while migrating only the issues content. For PRs, we put a link to the original PR in Pagure.

In case of private issues, or deleted PRs or issues, there will be gaps between the IDs. We have decided to fill these gaps only with empty issues.

Pull Requests migration

But if someone wants to migrate PRs, yes it can be done, it’s just a bit tricky. To migrate PRs with their diff, you need to pull the last commit from Pagure before the pull request and the last commit of the pull request. This would be something like this:

# you are in a main branch
git switch -c pr-branch
git reset --hard the-last-commit-of-pr
git switch -c main-before-pr
git reset --hard the-last-commit-before-pr
# do your changes in <pr-branch>
git push <remote> <pr-branch>:<main-before-pr>

This pushes all changes to the main-before-pr branch, which you can create PR in GitHub (or via GitHub API) and then merge PR and delete the branch immediately. The PR history will remain.

Rate limits

Then GitHub stabbed me in the back with its crazy limits. According to their instructions, I did the sleep(1) between POST requests, but after about 40 requests I always hit the limit anyway, and for this case in there is no Retry-After parameter in the header. With each subsequent hit of the limit, there was a longer and longer punishment, even tens of minutes. At this point, I gave up and just put sleep for 2 minutes between each POST request. Also, I shrunk the issue content just to one long message so that it was just one POST request example. So we’ll be migrating for 4 days…

Maybe I could get around these limits by running the script as a GitHub Action, on which there are other limit rules, but I didn’t want to waste my time with that.

The migration process itself

The migration itself wasn’t that complicated - from a code point of view.

I created API keys for Pagure and GitHub

Downloaded the Pagure data into a JSON file for good measure (so that nothing would change during the long 4-day migration) source

Then just convert the data back from the JSON file:

  pagure_prs = {
      pr.id: pr
      for pr in [
          ogr.services.pagure.PagurePullRequest(pr_dict, self.pagure_project)
          for pr_dict in pagure_prs_json
      ]
  }

Incremental migration of Copr
- First I migrated without any content. script at that time
- Then I migrated the labels. script at that time
- After a couple of weeks, I migrated the issue’s content as well (that’s why the code isn’t logically coherent). script at that time
On Pagure in Project Options we set the issue tracker to read-only.

Conclusion

I recommend using ogr for migration, it saved me some nerves due to the fact that it wraps Pagure API in Python and it also provides functionality to migrate data from Pagure to GitHub. It just requires a lot of patience, because of the very strict limits on the GitHub side, but if you download the data from Pagure before the migration it can run for a really long time without consequences.

Highlights from 2023-01-25 release

Copr Team — Wed, 25 Jan 2023 00:00:00 +0000

Copr - look back at 2022

Miroslav Suchý — Mon, 09 Jan 2023 00:00:00 +0000

Let me sum up what the Copr team did during 2022.

Mock:

We did six releases of Mock Starting with a major upgrade to 3.x that dropped python2 support and EL7 as the host platform.
We added --list-chroots option. Allow better customization of used tar binary and adapt to the new split of qemu-user-static package.
We also added lots of new chroots: AlmaLinux, RockyLinux, EuroLinux, OpenEuler, and a few others…

Copr:

We did nine releases of Copr and upgraded Copr servers to Fedora 37.
We wrote two “4 cool new projects to try in Copr” articles for Fedora Magazine

Beside previously built of all gems from Rubygems.org we built all modules from PyPI as RPM packages [Thousands of PyPI and RubyGems RPMs now available for RHEL 9

Red Hat Developer](https://developers.redhat.com/articles/2022/06/07/thousands-pypi-and-rubygems-rpms-now-available-rhel-9) Big thanks to Karolina Surma from Python team on cooperation on this. As a side effect, we introduced pyp2spec as a second option to build directly from PyPI.

We presented at Fedora Nest.
We dropped APIv2. And provided guidance how to migrate your scripts https://fedora-copr.github.io/posts/api3-migration-helper
We added Kerberos authentication to command line tools and API https://fedora-copr.github.io/posts/how-to-use-kerberos-in-copr
We cooperated with Packit on building SRPM in Copr https://packit.dev/postcs/copr-srpms/
We started using IBM Cloud for native s390x builders https://pavel.raiskup.cz/blog/fedora-copr-uses-ibm-cloud.html
As a side effect we packaged python modules for managing resources in IBM Cloud.
We spent lots of time optimizing the scheduler in Copr. E.g.
- Builds from webhooks are now background jobs https://docs.copr.fedorainfracloud.org/release-notes/2022-02-03.html#webhook-rebuilds-are-background-jobs-now
- We improved the throughput when the queue is bigger than 70k jobs https://docs.copr.fedorainfracloud.org/release-notes/2022-03-21.html#large-queue-improvements
- And we were able to increase quota of parallel builds from one user from 35 to 45.
We started using SHA256 for signing packages https://docs.copr.fedorainfracloud.org/release-notes/2022-03-21.html#signing-packages-with-sha256
We started using OpenPGP v4 signatures and we were one of the first to discover issues with new Sequoia backend of RPM with older version of signatures.
We created a webUI statistics page that shows the utilization of resalloc resources https://docs.copr.fedorainfracloud.org/release-notes/2022-06-22.html#resalloc-webui
We (finally) were able to count download statistics from CDN https://docs.copr.fedorainfracloud.org/release-notes/2022-08-18.html#rpm-download-statistics
You can submit more builds at once from command line https://docs.copr.fedorainfracloud.org/release-notes/2022-07-27.html#submitting-multiple-builds-at-once-via-copr-cli
We upgraded aarch64 builders to stronger Graviton3 machines https://docs.copr.fedorainfracloud.org/release-notes/2022-11-28.html#updated-aarch64-builders-to-graviton3-processors
We migrated our git project to GitHub https://docs.copr.fedorainfracloud.org/release-notes/2022-11-28.html#development-moved-to-github
We migrated to new storage as we hit 16 TB hard limit for one volume in AWS. It took us few weeks of preparation to make just a few hour outage for building packages and almost no outage for yum repositories availability.
Fedora Infrastructure helped us to provision a new Power9 box (for ppc64le builds) which hosts 25 builders (which complements the set of Power9 machines in OSUOSL). Note that Power9+ is needed for Enterprise Linux 9 builds.
We created OpenShift and Kustomize deployment script.
Statistics:
- Copr run 1 251 633 builds (one package for different chroots is conted as one)
- People created 23 053 new projects.

Fedora:

We participated in SPDX License migration phase 1 https://fedoraproject.org/wiki/Changes/SPDX_Licenses_Phase_1
We announced the initial release of Package Review service https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/E4TT2PEOSITJ4PJP44L2GQUU4CA6R6B3/
We contributed to fedora-review to produce machine-readable output https://pagure.io/FedoraReview/pull-request/463
One modulemd-tools release.
FIXME another notable fedora-review contribution?

Others:

We did three releases of Tito
We did seven releases of Resalloc
RPM Spec Wizard got a nice front page https://xsuchy.github.io/rpm-spec-wizard/
Copr users are known to build the same NEVRAs multiple times in the same project, which confuses DNF and other tools. We contributed to createrepo_c https://github.com/rpm-software-management/createrepo_c/pull/325 which is the first step to deterministically resolve the problem.
The “differential PyLint checker” we used for the last few years was separated from the Copr base code into https://github.com/fedora-copr/vcs-diff-lint and it can now be easily used by other projects. We also provide a “glue” GitHub Action wich makes it’s use trivial, see integration example https://github.com/rpm-software-management/mock/blob/main/.github/workflows/python-diff-lint.yml

Outlook for 2023

Integration with Koshei - automatic rebuild of your package in your project when dependency changes (inherited from the previous year - not even started yet)
Hopefully start using ImageBuilder.
We are investigating the usage of Pulp as a backend for storing RPM packages.

Ideas we have for 2023 - some of them are inherited from the previous year:

Enhance Mock --chain to try to set %bootstrap when the standard loop fails. When the set succeeds, rebuild the bootstrapped package again without the %bootstrap macro.
Contribute to fedpkg/koji to have machine-readable output. FIXME
Include packages from Copr in results of https://packages.fedoraproject.org/
Allow running various tools right after the build - e.g., rpminspect, swidtags.
Automatically verify if your package’s license can be used in Copr.

The Community Packaging Team consists of Pavel Raiskup, Jakub Kadlcik, Jiří Kyjovský (who replaced Silvie Chlupova), and me.

Highlights from 2022-11-30 release

Copr Team — Wed, 30 Nov 2022 00:00:00 +0000

Highlights from 2022-11-28 release

Copr Team — Mon, 28 Nov 2022 00:00:00 +0000

Nest 2022 — Fedora Copr Stories

Pavel Raiskup — Mon, 24 Oct 2022 00:00:00 +0000

Migrate to APIv3 queries

Pavel Raiskup — Thu, 22 Sep 2022 00:00:00 +0000

We had planned the APIv2 drop for a very long time, and we started with that quite some time ago (api_2 dropped from our Python API lib). The team was so much familiar with this ongoing change, and was kind of bored from announcements so we forgot about analyzing the ongoing api_2 Apache access_log entries.

The change has already happened, api_2 is gone. So here comes at least a small helper post that should make the migration from api_2 to api_3 trivial. Only the routes that are being accessed (and are 404) are covered here.

Also, it might be a good idea to use the Python API (python3-copr) if possible.

How to get a build info

Old route: /api_2/builds/<ID>
New route: /api_3/build/<ID>

This should be easy as it looks. But note that newly you want look at the .projectname and .ownername fields in the output because you’ll need them in the other routes.

How to get a project info

Old route: /api_2/projects/<project_id>
New route: /api_3/project?ownername=<ownername>&projectname=<projectname>

Listing builds in a project

Old route: /api_2/builds?project_id=<proj_id>
New route: /api_3/build/list/?ownername=<ownername>&projectname=<projectname>

We no longer work with project IDs here.

Note the needed trailing slash symbol after list/?..., this is a bug and should be fixed so both list? and list/? variants are possible in the future.

Last build in a project

Same as the previous one, just add the &limit=N argument:

Old route: /api_2/builds?project_id=<proj_id>&limit=<N>
New route: /api_3/build/list/?ownername=<ownername>&projectname=<projectname>&limit=<N>

List of owner’s projects

Old route: /api_2/projects?owner=<owner>&name=<owner>
New route: /api_3/project?ownername=<owner>&projectname=<projectname>

Build config

The “build config” info is rather internal thing (we have part of it in APIv3, because of the $ copr mock-config <project> <chroot> functionality. But previously the config route was used to get the build result directory, aka chroot’s result_dir_url. This can be now obtained through the /api_3/build-chroot route (result_url field). The rest of the info goes to /backend internal namespace.

Old route: /api_2/build_tasks/<build_id>/<chroot_name>
New route: /api_3/build-chroot?build_id=<build-id>&chrootname=<chroot_name>
New route: /api_3/build-chroot/build-config?build_id=<build-id>&chrootname=<chroot_name>
New route: /backend/get-build-task/<build-id>-<chroot_name> (internal only)

Highlights from 2022-09-21 release

Copr Team — Wed, 21 Sep 2022 00:00:00 +0000

4 cool new projects to try in Copr for August 2022

Jiří Kyjovský — Mon, 22 Aug 2022 00:00:00 +0000

Highlights from the 2022-08-18 release

Copr Team — Thu, 18 Aug 2022 00:00:00 +0000

Highlights from 2022-07-27 release

Jakub Kadlčík — Wed, 27 Jul 2022 00:00:00 +0000

4 cool new projects to try in Copr for May 2022

Miroslav Suchý — Fri, 27 May 2022 00:00:00 +0000

Why my patches are missing from custom build

Ding-Yi-Chen — Tue, 19 Apr 2022 00:00:00 +0000

How to use Kerberos in Copr

Silvie Chlupova — Thu, 07 Apr 2022 00:00:00 +0000

During the last release, a new authentication option was added to Copr. Now users can also log in using a Kerberos ticket. We use GSSAPI to implement this mechanism. The GSSAPI (Generic Security Services API) is a common interface for accessing the Kerberos security system. Enabling GSSAPI authentication in Copr CLI or API will allow authentication without using the API token for clients that have a Fedora account. To use GSSAPI login in WebUI, click on gssapi-login (negotiate needs to be allowed for the .fedorainfracloud.org domain in the browser, see browser configuration). It will log you in if you have a Kerberos ticket for username@FEDORAPROJECT.ORG. To enable GSSAPI, you need to obtain a Kerberos ticket first. If you want to work with Fedora Copr, you can just do:

$ fkinit # optionally with "-u username"
Enter your password and OTP concatenated. (Ignore that the prompt is for only the token)
Enter OTP Token Value: <your password + OTP token>
$ copr whoami
your_username

You can also find here how to use Kerberos auth with Fedora Infrastructure. To work with another (non-Fedora Copr) instance, you need to obtain the Kerberos ticket differently, and you need to have a configuration file containing the appropriate copr_url:

$ kinit username@EXAMPLE.COM
$ cat ~/.config/copr
[copr-cli]
copr_url = https://copr.example.com/

API key vs. Kerberos

If you have an API key set up and have a Kerberos ticket, then the API key has a higher priority and will be used. By default, we enable the use of GSSAPI in the CLI. This means that if you have a properly configured API key, it will be used, but if not, we try to use GSSAPI to get the necessary information. Be careful, in this case, CLI will use the copr_url set up in the configuration file to get the information, and if you don’t have the configuration file at all, it will use the default URL, which is https://copr.fedorainfracloud.org. If you want to use another instance of Copr, set copr_url to the preferred instance. You can disable the use of GSSAPI by setting gssapi to False in the configuration file.

[copr-cli]
gssapi = False

Unlike CLI, this behavior is disabled by default in the API. However, you can still use the method create_from_config_file.

from copr.v3 import Client
client = Client.create_from_config_file()
print(client.config)

{'username': None, 'login': None, 'token': None, 'copr_url': 'https://copr.fedorainfracloud.org', 'gssapi': None, 'encrypted': True}

You can see that gssapi is not set so anything you do now using the API and requires authentication will fail and throw an exception CoprAuthException. You can solve this by setting:

client.config["gssapi"] = True

Switch primary ticket cache

If you obtain a ticket, e.g., for username@EXAMPLE.COM, and then you obtain a ticket for username@FEDORAPROJECT.ORG, everything works well. However, if you obtain a ticket first for FEDORAPROJECT.ORG and then for EXAMPLE.COM, it will not be possible to log in to the WebUI or use CLI and API. You can solve this issue by using kswitch.

kswitch -p username@FEDORAPROJECT.ORG

it makes the specified credential cache the primary cache for the collection if a cache collection is available. No matter the order in which you get the ticket now, when you list cached Kerberos tickets using klist

klist -A

you can see that the Kerberos ticket for username@FEDORAPROJECT.ORG is now listed as first.

New users

If you are a new user and want to start using Copr or if your group membership changed, and you want to use this new group in Copr, you will first need to log in to the WebUI using the log in link. We are currently unable to retrieve group membership and your email from FAS when you log in using Kerberos. Behind the log in link is openid, which can do this. We need this information about new users in order to use Copr.

Logic behind it

Highlights from 2022-04-05 release

Pavel Raiskup — Tue, 05 Apr 2022 00:00:00 +0000

Highlights from 2022-03-21 release

Jakub Kadlčík — Mon, 21 Mar 2022 00:00:00 +0000

Packit - Building SRPMs in Copr

Laura Barcziova — Tue, 15 Mar 2022 00:00:00 +0000

Building RHEL on a subscribed Fedora cloud box

Pavel Raiskup — Thu, 10 Mar 2022 00:00:00 +0000

IBM Cloud used in Fedora Copr

Pavel Raiskup — Mon, 14 Feb 2022 00:00:00 +0000

Highlights from 2022-02-03 release

Silvie Chlupova — Thu, 03 Feb 2022 00:00:00 +0000

Copr - look back at 2021

Miroslav Suchý — Thu, 20 Jan 2022 00:00:00 +0000

Let me sum up what the Copr team did during 2021

Mock

We did eight releases of Mock.

We moved Mock’s wiki to GitHub Pages to allow indexing by search engines https://rpm-software-management.github.io/mock/ and created a Fedora-based Jekyll container for local documentation testing (https://github.com/praiskup/jekyll-github-pages-fedora-container).

We initiated the discussion about default epel-8-* config

Copr

We did six releases of Copr and upgraded Copr servers to Fedora 35.

We wrote three “4 cool new projects to try in Copr” articles for Fedora Magazine.

We rebuilt all gems from Rubygems.org for Fedora Rawhide.

We started to use AWS Spot instances for builders.

We started to decommission APIv1 and APIv2.

You have an option to run a fedora-review after each build.

We created a new Ansible module copr which is available in community general collection

You can order your builds using batches now

People started using discussion under projects. There are more than one hundred active discussions https://discussion.fedoraproject.org/c/projects-in-copr/54

We redesigned Copr’s home page.

We worked on clean-up scripts resulting in 5+ TB cleaned from our backends.

We did six releases of resalloc with improvements for better throughput and reactions during peeks. https://github.com/praiskup/resalloc/

Copr’s servers got IPv6

We did three releases of prunerepo https://pagure.io/prunerepo

We added lots of builders and some architectures and later we re-add ppc64le architecture, and started using spot AWS instances

We implemented Error Budget and our goal is:

97 % of builds of copr-ping package is finished within 6 minutes (this monitor length of queue and speed of builders)
99,3 % uptime of CDN
99,3 % uptime of copr-backend (dnf repositories) (cca 5h/month)
97.5 % uptime of copr-frontend (WebUI) (cca 18h/month)

There is work in progress on Kerberos authentication in copr-cli.

Statistics:

Copr run 2,900,000 builds.
People created 15 731 new projects.

Fedora

We created a Fedora Sponsor site to easy find of a sponsor.

We created a video explaining a dist-git.

We proposed Retired Packages change and it got accepted.

We created license-validate tool.

Others

We did four releases of Tito.

We wrote an article about activating no-cost RHEL.

We wrote three articles about storing GPG keys in DNS and persuaded several distributions to put the records in DNS https://github.com/xsuchy/distribution-gpg-keys/#storing-keys-in-dns

New modulemd-tools release with “bld2repo” tool.

Outlook for 2022

We are in the middle of talking with IBM, which should result in the availability of native s390x builders in the early months of 2022

We had an initial meeting about rebase-helper automatically opening PR in src.fedoraproject.org. There is even some code written (by Michal Konečný), but the code is not integrated yet and no user-visible outcome was done yet. ETA is the first half of 2022.

Python team deprecated pyp2rpm and Karolina Surma is writing a new tool from scratch and she will use it for the rebuild of PyPI in Copr in a similar way to how we have done Rubygems. ETA is early months or 2022.

Unify forge events - when an “interesting” event happens on GitHub/GitLab/* sent notification to Fedora Messaging in a unified format. Besides Copr, this will be useful for Packit too.

s we have for 2022 - some of them are inherited from the previous year:

Finish rpm-spec-wizard https://github.com/xsuchy/rpm-spec-wizard

Integration with Koshei - automatic rebuild of your package in your project when dependency change

Enhance Mock --chain to try to set %bootstrap when the standard loop fails. When the set succeeds, rebuild the bootstrapped package again without the %bootstrap macro.

Contribute to fedpkg/koji to have machine-readable output.

Building VM images for Cloud (osbuild-composer)

If you have any idea which can ease packaging (especially the automation) then do not hesitate and share it with us. We may do that!

The Community Packaging Team consists of Pavel Raiskup, Silvie Chlupova, Jakub Kadlcik, and Miroslav Suchý.

Curious what we did in 2020?

4 cool new projects to try in COPR for December 2021

Jakub Kadlčík — Mon, 20 Dec 2021 00:00:00 +0000

Highlights from 2021-11-11 release

Silvie Chlupova — Thu, 11 Nov 2021 00:00:00 +0000

Rebuilding the entire RubyGems in Copr

Jakub Kadlčík — Tue, 09 Nov 2021 00:00:00 +0000

Highlights from 2021-10-01 release

Silvie Chlupova — Fri, 01 Oct 2021 00:00:00 +0000

AWS spot/on-demand Copr builders

Pavel Raiskup — Mon, 02 Aug 2021 00:00:00 +0000

4 cool new projects to try in Copr from July 2021

Jakub Kadlčík — Fri, 30 Jul 2021 00:00:00 +0000

Fedora Copr farm of builders - status of July 2021

Pavel Raiskup — Tue, 13 Jul 2021 00:00:00 +0000

Highlights from 2021-06-16 release

Pavel Raiskup — Wed, 16 Jun 2021 00:00:00 +0000

EOL Copr APIv1 and APIv2, pt.2

Jakub Kadlčík — Sun, 09 May 2021 00:00:00 +0000

During the last year, we are incrementally dropping support for Copr’s APIv1 and APIv2. We kindly ask you to migrate to APIv3. Some reasoning and our motivation for doing so can be found the Copr has a brand new API blog post.

According to the deprecation schedule, we just took the following step:

April 2021 - print warning on STDERR when you use APIv2 from python3-copr and remove APIv1 from python3-copr library

That means that if you cannot migrate to APIv3 yet, you can still send requests to APIv1 endpoints directly, without using the python3-copr library, or you can temporarily freeze on python-copr-1.109 and copr-cli-1.93.

Please don’t put aside the migration of your code for much longer. In September the APIv1 is going to be dropped from the frontend and APIv2 from clients.

Highlights from 2021-04-28 release

Jakub Kadlčík — Wed, 28 Apr 2021 00:00:00 +0000

Running fedora-review after Copr build

Jakub Kadlčík — Sun, 18 Apr 2021 00:00:00 +0000

4 cool new projects to try in Copr for March 2021

Jakub Kadlčík — Fri, 26 Mar 2021 00:00:00 +0000

Copr release 21.03

Miroslav Suchý — Thu, 18 Mar 2021 00:00:00 +0000

We have deployed the new version of Copr. You can read full release notes.

The highlights are:

Automatic run of Fedora Review tool
The centos-stream-8 chroot change

New Ansible module Copr

Silvie Chlupova — Wed, 03 Feb 2021 00:00:00 +0000

We created a new Ansible module for Copr, which was added to the community general collection in Ansible galaxy during the release of the latest version 2.0.0. The module code can be found in the GitHub repository of the mentioned collection.

Synopsis

This module can enable, disable or remove the specified repository.

Quick setup

To install a collection hosted in Galaxy:

ansible-galaxy collection install community.general

You can also install the ansible-collection-community-general package with our module as soon as the Fedora package version is updated to version 2.0 (currently available version is 1.0):

dnf install ansible-collection-community-general

Or you can use the tarball (version 2.0.0):

ansible-galaxy collection install https://github.com/ansible-collections/community.general/archive/2.0.0.tar.gz -p ./collections

Module usage

To use it in a playbook, specify: community.general.copr.

Examples of use in a playbook

Use state enabled to enable the Copr project, then use the name of the user/project and the chroot you wish to enable.

- name: Enable project Test of the user schlupov
  community.general.copr:
    host: copr.fedorainfracloud.org
    state: enabled
    name: schlupov/Test
    chroot: fedora-31-x86_64

Use state absent to remove the Copr project. The host is optional and copr.fedorainfracloud.org is used as the default. Likewise, chroot is not required, so if you use the module on fedora 32 and the x86_64 architecture, fedora-32-x86_64 will be used as chroot option.

- name: Remove project copr of the group copr
  community.general.copr:
    state: absent
    name: @copr/copr

Use state disabled to disable the Copr project. The repository can be seen in the /etc/yum.repos.d/ directory, but is set to disabled. On the other hand, in case of state absent, the repo file is completely removed from the system.

- name: Disable project copr of the group copr
  community.general.copr:
    state: disabled
    name: @copr/copr
    chroot: fedora-32-x86_64

Parameters

Parameter	Choices/Defaults	Comments	Required
host	Default: “copr.fedorainfracloud.org”	The Copr host to work with	No
protocol	Default: “https”	This indicate which protocol to use with the host	No
name		Copr project name, for example @copr/copr-dev	Yes
state	Default: “enabled”, Choices: “enabled”, “disabled”, “absent”	Whether to set this project as enabled, disabled or absent	No
chroot	Default: your “os-os_version-architecture”	The name of the chroot that you want to enable/disable/remove in the project, for example epel-7-x86_64. Default chroot is determined by the operating system, version of the operating system, and architecture on which the module is run	No

Order your Copr builds using batches

Pavel Raiskup — Mon, 04 Jan 2021 00:00:00 +0000

4 cool new projects to try in COPR from December 2020

Jakub Kadlčík — Mon, 21 Dec 2020 00:00:00 +0000

4 cool new projects to try in COPR from October 2020

Jakub Kadlčík — Mon, 02 Nov 2020 00:00:00 +0000

Submit Copr builds by GitHub “push” Actions

Pavel Raiskup — Thu, 03 Sep 2020 00:00:00 +0000

Upvoting projects in Copr

Jakub Kadlčík — Tue, 18 Aug 2020 00:00:00 +0000

EOL Copr APIv1 and APIv2

Miroslav Suchý — Tue, 04 Aug 2020 00:00:00 +0000

During Copr history, we got three APIs. For a long time, we maintained all versions.

We decided that it is time to remove the old versions. We are going to start with APiv1.

The schedule is:

September 2020 - print warning on STDERR when you use APIv1 from python3-copr.
April 2021 - print warning on STDERR when you use APIv2 from python3-copr and remove APIv1 from python3-copr library
September 2021 - remove APIv1 code from the frontend. I.e., Copr service will not understand APIv1 calls. Remove APIv2 from python3-copr library.
April 2022 - remove APIv2 code from the frontend. I.e., Copr service will not understand APIv2 calls.

I strongly encourage you to migrate your scripts to APIv3.

Deleting Multiple Builds in Copr

Dominik Tureček — Tue, 23 Jun 2020 00:00:00 +0000

Previously, when you wanted to delete lots of build in the Copr, you had to do something like:

for id in 10 12 13 15; do
    copr-cli delete-build $id
done

This has been no problem for small projects. But it was a big issue for projects with thousands of packages (e.g., python rebuilds). After each delete-build we run createrepo_c to update repository. And for these big projects, it can run for several minutes. It can easily take a day to delete hundreds of builds this way.

Since the copr-cli-1.87 (going to be released this week), you are now able to delete a group of builds specified by their IDs with a single command in the following style:

copr-cli delete-build 10 12 13 15

This generates one action for the backend, deletes all the builds at once and it calls createrepo_c only once. This leads to a much shorter time needed for deleting larger numbers of builds.

Runtime dependencies in Copr

Dominik Tureček — Tue, 02 Jun 2020 00:00:00 +0000

In the last release, we’ve added support for runtime dependencies in Copr. If your project needs another package to run, but you don’t need said package as a part of your project, you can have it as a runtime dependency instead. You can now specify other repositories (either Copr projects or external) as runtime dependencies. These repositories will then be enabled alongside your Copr project.

We are currently testing the feature and are looking for users’ feedback - please share any issues that you might find. Later, once this feature is tested, we plan to generalize it and add it to dnf and createrepo_c as well.

How to add runtime dependencies to your Copr project

To add runtime dependencies to your Copr project, navigate to your project’s Settings/Project Details tab. There, in Other options, is a field Runtime dependencies where you can specify a space-separated list of dependencies:

To specify another Copr project as a runtime dependency, add copr://user-name/project-name.
To specify an external repository as a runtime dependency, add a URL to its repo file.

You can specify as many runtime dependencies as you want. Note that the runtime dependencies are resolved transitively, e.g., if your ProjectA depends on ProjectB and ProjectB depends on ProjectC, then ProjectA will have both ProjectB and ProjectC as its dependencies.

Enabling a project with runtime dependencies

So, if I enable a Copr project that has a bunch of dependencies, how do I know what repositories I actually enabled? First, you can see a list of all runtime dependencies on the project’s Overview tab:

Second, if you download a repofile of such a project, it lists all the dependencies. And last, if you are using the dnf copr plugin for enabling Copr projects (using the dnf copr enable command), it will notify you about all the dependencies. Although the dependencies come enabled by default, the dnf copr plugin gives you an option to disable them.

4 cool new projects to try in COPR for May

Dominik Tureček — Mon, 04 May 2020 00:00:00 +0000

Proxies for Copr repositories

Miroslav Suchý — Tue, 04 Feb 2020 00:00:00 +0000

I just enabled the content delivery network (CDN) for Copr repositories. It is provided by CloudFront from AWS. And it is provided for free by Amazon to Fedora.

Technically the original URL

copr-be.cloud.fedoraproject.org

is now accessible using

download.copr.fedorainfracloud.org

The original URL is and will be available. You can still use it. The second one is CNAME for the CDN and should be much faster for you.

The CDN is automatically enabled for new projects. Though, if you already have enabled some copr repository on your workstation/server, it will still use the old URL. It is fine; it will continue to work. If you want to enable CDN for your repos you have to run:

dnf copr enable some/project

for every copr project you have enabled. Or manually change the URL in the repo file.

If you experience any problem with CDN, please let me know.

4 cool new projects to try in COPR for January

Dominik Tureček — Wed, 29 Jan 2020 00:00:00 +0000

4 cool new projects to try in COPR for October

Dominik Tureček — Fri, 25 Oct 2019 00:00:00 +0000

Copr's WebUI supports Kerberos

Miroslav Suchý — Tue, 08 Oct 2019 00:00:00 +0000

Copr ported to fedora-messaging

Pavel Raiskup — Tue, 10 Sep 2019 00:00:00 +0000

Create your portfolio with pinned projects

Jakub Kadlčík — Mon, 19 Aug 2019 00:00:00 +0000

4 cool new projects to try in COPR for August

Dominik Tureček — Mon, 05 Aug 2019 00:00:00 +0000

Parse log from failed build using Mock

Silvie Chlupova — Fri, 19 Jul 2019 00:00:00 +0000

Into mock has been added a new feature called mock-parse-buildlog.

This piece of code will help you identify what caused your build to fail. Currently, mock-parse-buildlog knows only two situations that could occur and cause that a build fail. First is that some files are in %buildroot but not in %files and the second one is that some files are in %files but not in %buildroot.

Part of the code has been taken from rebase-helper.

An example of how it works

Run as:

mock-parse-buildlog --path /path/to/log/build.log

And the result is:

Error type: Build failed because problematic files are in %buildroot but not
in %files

Problematic files:
/usr/share/locale/sv/LC_MESSAGES/hello.mo
/usr/share/locale/uk/LC_MESSAGES/hello.mo
/usr/share/locale/pt_BR/LC_MESSAGES/hello.mo
[...]

However, if you see an error it means that mock-parse-buildlog couldn’t recognize what caused that the build failed thus it cannot determine the problem and help you solve it. And this is the spot where other users of mock can help us. If you know that something once (or more times) caused that your build failed, please let us know by creating a Github issue because the same problem may occur with someone else. We can work together to improve this tool.

Copr's Dist-Git

Miroslav Suchý — Thu, 06 Jun 2019 00:00:00 +0000

Fedora use dist-git as the ultimate source for SPEC files.

In Copr, we use dist-git to store sources as well. However, our use case is different. In the past, Copr only allowed to build from URL. You provided a URL to your SRC.RPM and Copr downloaded it and built it. This was a problem when the user wanted to resubmit the build. The original URL very often did not exists anymore. Therefore we came with an idea to store the SRC.RPM somewhere. And obviously, the dist-git was the first idea.

Though there is a big difference between Fedora’s dist-git and Copr’s dist-git. In Fedora, only the owner interacts with dist-git. He maintains the branches, addresses pull requests, resolves conflicts, etc. In Copr, it is different. The biggest user of Copr’s dist-git is Copr itself. Copr uploads new SRC.RPM - either those available from URL, or uploaded via API, or after creating the SRC.RPM from upstream git. While you can clone the dist-git, or see it via cgit, you are not allowed to write to your dist-git. It is internal storage for Copr.

This has one implication: if you want to send maintainer a patch for his SPEC file, please do not use the SPEC file from Copr’s dist git. The spec file is not the original source (for the maintainer of the project). It can be created on the fly - e.g. using pyp2rpm or gem2spec. You should always contact the owner of the project and ask him, what is his preferred way to contribute to the project.

Prioritize BuildRequires from Copr

Jakub Kadlčík — Sun, 28 Apr 2019 00:00:00 +0000

4 cool new projects to try in COPR for April

Dominik Tureček — Sat, 20 Apr 2019 00:00:00 +0000

Copr has a new RSS feed

Jan Buchmaier — Fri, 29 Mar 2019 00:00:00 +0000

Now you can keep track of every new Copr project thanks to our new feed.

What is the content of the feed

Our RSS feed contains two hundred most recent projects.

And in every item you can find:

author’s and project’s name
link to the project
simple description

Where to find it

You can visit our feed.

Copr - Removing outdated chroots

Jakub Kadlčík — Sun, 10 Mar 2019 00:00:00 +0000

4 cool new projects to try in COPR for February

Dominik Tureček — Mon, 11 Feb 2019 00:00:00 +0000

4 cool new projects to try in COPR for December

Dominik Tureček — Fri, 21 Dec 2018 00:00:00 +0000

4 cool new projects to try in COPR for October

Dominik Tureček — Tue, 30 Oct 2018 00:00:00 +0000

Starting your own distro

clime — Tue, 18 Sep 2018 00:00:00 +0000

Copr has a brand new API

Jakub Kadlčík — Wed, 08 Aug 2018 00:00:00 +0000

4 cool new projects to try in COPR for July

Dominik Tureček — Fri, 20 Jul 2018 00:00:00 +0000

4 cool new projects to try in COPR for June

Dominik Tureček — Mon, 04 Jun 2018 00:00:00 +0000

CI/CD with Copr, finally for any project?

Pavel Raiskup — Mon, 21 May 2018 00:00:00 +0000

4 cool new projects to try in COPR for April

Dominik Tureček — Tue, 17 Apr 2018 00:00:00 +0000

Modularity features in Mock

Jakub Kadlčík — Mon, 12 Mar 2018 00:00:00 +0000

Modules with Copr packages

Jakub Kadlčík — Thu, 08 Mar 2018 00:00:00 +0000

Prioritize BuildRequires from Copr

Jakub Kadlčík — Mon, 05 Mar 2018 00:00:00 +0000

How to build modules in Copr

Jakub Kadlčík — Wed, 28 Feb 2018 00:00:00 +0000

4 cool new projects to try in COPR for February

Dominik Tureček — Fri, 23 Feb 2018 00:00:00 +0000

4 cool new projects to try in COPR for January

Dominik Tureček — Wed, 24 Jan 2018 00:00:00 +0000

Copr Modularity in retrospect

Jakub Kadlčík — Fri, 12 Jan 2018 00:00:00 +0000

How to run a database in %check section of RPM spec file

Miroslav Suchý — Tue, 28 Nov 2017 00:00:00 +0000

Let the blogging begin!

Jakub Kadlčík — Tue, 21 Nov 2017 00:00:00 +0000

Welcome to the official blog for the Copr project. Its purpose is to keep you updated with new features, sharing how-to tutorials and other kinds of interesting information. Posts are written mainly by developers, but we encourage you to write and submit your posts too. A topic of an article can be absolutely anything that is related to Copr.

How to write a post

It is fairly easy to write a new post. The whole process is described in the README and explains how to create a post, how it should be formatted, how to run this page locally to view the changes and how to submit the pull request if everything looks as you wanted. If you miss any information, please contact us.

Technologies

This page is built on Jekyll, it’s source code is available on GitHub and is deployed via GitHub Pages. The comment section is provided by Disqus.

We produce two RSS feeds, one is called feed.xml and is aimed at users to subscribe. It contains all posts, even the ones that are hosted on external sites such as a personal blog of a particular developer. The second feed is called planet.xml and is subscribed by Planet Fedora. This one contains only posts hosted on this site.

Tell us your impressions in the comment section below.

Copr stack dockerized!

Jakub Kadlčík — Tue, 17 Oct 2017 00:00:00 +0000

Reproducible Copr builds

Jakub Kadlčík — Sat, 14 Oct 2017 00:00:00 +0000

Copr Rebuild Tools - Tito backend

Jakub Kadlčík — Fri, 14 Jul 2017 00:00:00 +0000

DistGit 1.0

Michal Novotný — Sat, 20 May 2017 00:00:00 +0000

Copr <3 Modularity

Jakub Kadlčík — Fri, 21 Apr 2017 00:00:00 +0000

Copr Rebuild Tools

Jakub Kadlčík — Mon, 15 Aug 2016 00:00:00 +0000

Copr - Threads support for dist-git

Jakub Kadlčík — Fri, 08 Jul 2016 00:00:00 +0000

Copr - Rubygems

Jakub Kadlčík — Sat, 28 May 2016 00:00:00 +0000

Copr CLI - Building from Tito, Mock and PyPI

Jakub Kadlčík — Mon, 23 May 2016 00:00:00 +0000

WIP - rebuilding all PyPi modules as RPM packages

Miroslav Suchý — Thu, 21 Apr 2016 00:00:00 +0000

New features in Copr

Miroslav Suchý — Tue, 29 Mar 2016 00:00:00 +0000

Copr - Forking Projects

Jakub Kadlčík — Sun, 13 Mar 2016 00:00:00 +0000

Copr statistics

Miroslav Suchý — Fri, 27 Mar 2015 00:00:00 +0000

How to get notification about your builds in Copr

Miroslav Suchý — Fri, 21 Mar 2014 00:00:00 +0000

Interesting projects in Copr

Miroslav Suchý — Thu, 13 Mar 2014 00:00:00 +0000