758 collectifs

Althing Web

A propos

Des sites WordPress rapides, sur-mesures et éco-conçus, connectés à CiviCRM ou à tout autre service tiers dont vous avez besoin !

Com 360 (Besançon)

A propos

Vos partenaires pour une communication à 360°

Hedwige

A propos

Entrez dans la magie du design

HackYourJob Community

A propos

L'association des développeuses et des développeurs freelance épanouis

focus.

A propos

sharper together — Stratégie, Marque, Naming, Communication, Social

Français en [langue]

A propos

Traductions du français en allemand, italien et/ou anglais.

esprit collectif

A propos

Nous sommes un collectif d’entrepreneurs et designers convaincus que l'intelligence collective et le design doivent accélérer la transition écologique. www.espritcollectif.fr

Neargem

A propos

Swiss Army Software Engineering

Formatrices et formateurs HackYourJob

A propos

Construction et animation de formations pour les professionnels de l'IT, par des faiseurs freelances ou assimilés, sur des hard skills et soft skills. Coaching, audit et conseil.

Corporate Voices

A propos

Le collectif qui vous aide à construire et alimenter votre stratégie pour servir votre marque corporate.

BeBranded

A propos

BeBranded vous accompagne dans la création de votre site internet et l'automatisation de vos process

Bumb.

A propos

Nous donnons à votre start-up un air de licorne

Ansea

A propos

Collaborative Commerce: combining Automation and Generative AI

Cliché

A propos

Designing the obvious

AMPERE

A propos

Le collectif spécialisé pour vos projets énergétiques.

LinkingBrand Digital Factory

A propos

+20% de ROI Digital. Grâce à 100 experts digitaux & aux méthodes des Digital Factory

La Lucarne

A propos

L'agence qui vous accompagne dans vos projets web

Updev Solutions

A propos

Booster votre productivité avec l'IA .

Carbonfact logo

Carbonfact

Mode & TextilePetite entreprise


Improving Data Import Efficiency

Implementing a background job platform and introducing parallelism at the application level to enhance performance and significantly reduce computing time.


Carbonfact is developing carbon management software for the fashion industry. The company imports and processes extensive data to provide comprehensive reporting and support brands in reducing their carbon footprint. However, some data imports took over 10 hours and required more than 250 GB of disk space, significantly slowing down the process. This delay left limited time for retries and prevented timely client feedback.

Process

  1. Setup BullMQ
    Introduced BullMQ to run data import jobs asynchronously with retry capabilities.

  2. Identify Slowness
    Conducted a comprehensive analysis to identify bottlenecks and rewrote slow parts of the data import process.

  3. Implement Streaming Approach
    Redesigned the import script to utilize a streaming approach, significantly reducing disk space and memory usage.

  4. Leverage Parallelism
    Rewrote the script to take advantage of parallel processing and fully utilize all available machine cores.

Impact

  • Job Monitoring
    The engineering team can now monitor their jobs through existing tools like Sentry.

  • Easy Job Creation
    Engineers can easily create new jobs using the provided documentation and SDK.

  • Processing Time Reduction
    Reduced data import time from 10 hours to just 30 minutes.

  • Disk Space Reduction
    Reduced disk space usage from 250 GB to just 1 GB.

  • Memory Usage Optimization
    Capped memory usage at 4 GB.

By implementing these changes, Carbonfact successfully streamlined their data import process, improved efficiency, and enhanced customer satisfaction by delivering quicker and more accurate insights.

Membres

Expertises

Google Cloud PlatformNestNode