When to Use

You need to find relevant or highly cited papers on a topic using Google Scholar or PubMed.
You have identifiers (e.g., DOI, PMID, arXiv ID, URL) and must convert them into correct BibTeX.
You want to verify citation accuracy (DOI resolution, required fields, consistency with CrossRef/PubMed).
You need to clean, deduplicate, sort, and standardize an existing .bib file before submission.
You are preparing a thesis/manuscript and need a reproducible workflow from search → extraction → formatting → validation.

Key Features

Paper discovery
- Google Scholar search with year filtering, pagination, and citation-count sorting.
- PubMed search with MeSH terms, field tags, publication-type filters, and date ranges.
Metadata extraction
- Resolve DOI/PMID/arXiv/URL to structured metadata via CrossRef, PubMed E-utilities, and arXiv APIs.
- Batch processing from files containing mixed identifiers.
BibTeX generation & cleanup
- Generate BibTeX entries with appropriate entry types and required fields.
- Format, sort (key/year/author), and deduplicate BibTeX libraries.
Citation validation
- DOI resolution checks and metadata cross-checking.
- Required-field checks by entry type, syntax validation, duplicate detection, and optional auto-fix.
Workflow integration
- Produces submission-ready .bib files for LaTeX/Overleaf workflows and complements literature review pipelines.

Dependencies

Python: 3.10+ (recommended)
Python packages:
- requests>=2.31.0
- scholarly>=1.7.11 (optional; required only for Google Scholar automation)

Example Usage

A complete, end-to-end workflow that searches, extracts metadata, formats, deduplicates, and validates a bibliography:

# 1) Search PubMed (biomedical focus)
python scripts/search_pubmed.py \
  --query '"CRISPR-Cas Systems"[MeSH] AND "Gene Editing"[MeSH]' \
  --date-start 2020-01-01 \
  --date-end 2024-12-31 \
  --limit 200 \
  --output crispr_pubmed.json

# 2) Search Google Scholar (broad coverage)
python scripts/search_google_scholar.py "CRISPR gene editing therapeutics" \
  --year-start 2020 \
  --year-end 2024 \
  --limit 100 \
  --output crispr_scholar.json

# 3) Extract metadata from search outputs (or mixed identifiers)
cat crispr_pubmed.json crispr_scholar.json > combined_results.json
python scripts/extract_metadata.py \
  --input combined_results.json \
  --output combined.bib

# 4) Add known papers by DOI (append)
python scripts/doi_to_bibtex.py 10.1038/s41586-021-03819-2 >> combined.bib
python scripts/doi_to_bibtex.py 10.1126/science.aam9317 >> combined.bib

# 5) Format + deduplicate + sort (newest first)
python scripts/format_bibtex.py combined.bib \
  --deduplicate \
  --sort year \
  --descending \
  --output formatted.bib

# 6) Validate + auto-fix common issues + emit report
python scripts/validate_citations.py formatted.bib \
  --auto-fix \
  --report validation.json \
  --output final_references.bib

# 7) Inspect validation results
cat validation.json

Implementation Details

1) Search (Discovery)

Google Scholar (scripts/search_google_scholar.py)
- Supports query operators such as exact phrases ("deep learning"), author filters (author:LeCun), title-only (intitle:"neural networks"), exclusions (-survey), and year ranges.
- Typical parameters:
  - --year-start, --year-end: constrain publication years
  - --limit: cap results
  - --sort-by citations: prioritize highly cited papers (when supported by the script)
PubMed (scripts/search_pubmed.py)
- Uses NCBI E-utilities (e.g., ESearch/EFetch/ESummary) to retrieve PMIDs and metadata.
- Typical parameters:
  - --query: supports MeSH terms, field tags, and Boolean logic
  - --date-start, --date-end: publication date filtering
  - --publication-types: e.g., Clinical Trial,Review
  - --format: JSON or BibTeX output (if supported)

(See: references/google_scholar_search.md, references/pubmed_search.md)

2) Metadata Extraction (Normalization)

Identifier inputs: DOI, PMID, arXiv ID, URL, or mixed lists/files.
Primary sources:
- CrossRef API for DOI-centric journal metadata
- PubMed E-utilities for biomedical records (PMID/PMCID, MeSH, abstracts)
- arXiv API for preprints and versioned records
- DataCite API for datasets/software DOIs (if implemented/used)
Field mapping goals:
- Required: author, title, year
- Articles: journal, volume, number, pages, doi
- Conferences: booktitle, pages
- Preprints: repository + identifier (e.g., eprint, archivePrefix)

(See: references/metadata_extraction.md)

3) BibTeX Formatting (Quality & Consistency)

Entry types commonly produced: @article, @inproceedings, @book, @misc.
Formatting rules enforced/encouraged:
- Page ranges use -- (e.g., 123--145)
- Protect capitalization in titles with braces (e.g., {CRISPR})
- Consistent author formatting (Last, First and Last, First)
- Stable citation keys (project convention; often FirstAuthorYearKeyword)

(See: references/bibtex_formatting.md)

4) Validation (Correctness)

Validation typically checks:

DOI validity: resolves via doi.org and matches CrossRef metadata.
Required fields: present per entry type; no empty critical fields.
Consistency: year format, numeric volume/issue, page-range syntax, URL accessibility.
Duplicates: same DOI, near-identical titles, or same author/year/title combinations.
BibTeX syntax: braces/quotes, commas, unique keys, special character handling.

Outputs may include a machine-readable report (e.g., JSON) with errors and warnings. (See: references/citation_validation.md)

Citation Management