[FEATURE REQUEST] Syntax for citations

[jghauser]

Issues

• I have checked existing issues and there are no existing ones with the same request.

Feature description

Markdown ([@SomeAuthor2021]) and orgmode ([cite:@SomeAuthor2021]) both have syntax for citations. This allows for proper referencing in academic texts and enables the automatic generation of bibliographies. Having something similar in neorg would allow writing academic texts with neorg and, once pandoc exports are implemented, the generation of properly styled pdfs (well, syntax for footnotes might be needed first, too).

As the details are quite tricky (e.g. page numbers, citations with or without author name, and much much more), I think the best way of implementing this is to follow one of the existing standards (i.e. orgmode, pandoc markdown). Using one of these well thought-out standards will make sure that neorg accounts for the various needs of researchers regarding citations.

Given that pandoc will be used for converting from .norg to other file formats, what would probably be simplest is to take on pandoc's citation syntax, so that the relevant parts of the text can simply be passed on to it. In that case, what would be needed would be the following:

1. A syntax that indicates a citation (e.g. [cite:CITATION])

• Where CITATION would then just be whatever is accepted by pandoc.
• Note that the @ in CITATION would need to be retained as citations of the type -@SomeAuthor2021 also exist in pandoc (the - indicates that that only the year should be displayed for example when generating pdfs).

2. A way to specify the file containing the bibliographic data (e.g. as part of @document.met?)

• This needs to be passed to pandoc, so that it knows where to find the relevant citation entries
• Pandoc supports BibLaTex, BibTeX, CSL JSON, CSL YAML files

3. A way of specifying the Citation Style Language (.csl) file

• It defines the citation style to be used (see the pandoc manual).
• Again, this should be passed on to Pandoc.

Additionally, it might be nice to have:

• A way of adding bibliographic data within the .norg file (instead of in a separate file)

I'm not very proficient in lua (or programming, for that matter), and I think implementing this might be beyond my abilities. However, I would love to see this feature and would be willing to take a stab at implementing it.

Help

Yes, but I don't know how to start. I would need guidance

Implementation help

• I think first thing to think about would be what is an appropriate syntax. I don't think I have the required general overview to know what is important here.
• And then I guess, it would be good to know where things will need to be changed in order to implement this. Probably it would make sense to wait with this until the pandoc export functionality is working.

[mrossinek]

Hi @jghauser! Thanks for the suggestion! You happen to have a very coincidental timing as we are just starting to discuss some syntax changes to the general way of how links work (#178 (comment)).
Over on Discord, I just put forward a proposal for this. As mentioned by @vhyrro in the comment I linked to above, we will showcase those changes in the near future. In the meantime, here is my current proposal for how footnotes could work. I am also envisioning that these could be used for citations too:

 ** Footnotes
    I would like to introduce a syntax specifically for footnotes
    (there was some discussion around this at some point in the past,
    but I was unable to find a source for it).

    Here is my proposed syntax:
    @code norg
    ^ My footnote title
    The content of my footnote

    This is no longer part of the footnote.

    ^^ My multi-paragraph footnote title
    I can write multiple paragraphs here.

    Thus, the syntax of footnotes is identical to the definition list syntax.
    ^^
    @end
    I could then simply link to these footnotes using their title.

    This can also be nicely exploited for bibliographies:
    @code norg
    ^ Author2021
    = Bibtex references.bib Author2021
    @end
    where I sneakily envisioned a `Bibtex` insertion (or any other coupling tool).

    One even greater approach would be:
    @code norg
    = Bibtex references.bib
    @end
    which could automatically understand the bibtex labels to be link targets 🤯
    
    In whatever link syntax we end up with, this could then be cited within the text like so:
    - Old format example [citation text](^ Author2021)
    - Currently proposed new link format {^ Author2021}

[jghauser]

Great to hear that this is in the works!

I'm not sure if I understand all aspects of your proposal correctly, so please forgive me if the following small comments miss their mark.

• It would definitely be very useful to have a way of 'importing' all Bibtex labels from a file as link targets (which I think is the very last part of your proposal). Doing it manually for each entry would become extremely tedious very quickly.
• Where exactly would this = Bibtex references.bib be in a document? Are there any reasons for not having it in @document.meta? I don't have strong preferences here, I'm just trying to understand the proposal
• Please make sure to consider the possibility of there being multiple .bib files. This is definitely something that I've used in the past, and I'm sure others have too.
• I would consider retaining the '@' of the labels. This is standard in most current software, would avoid clashes with 'normal' footnotes, would make it easier to see which things are 'proper' footnotes and which are citations, and would make converting with Pandoc easier (see my comments in the first post). However, I do see the merit of not being too tied down to old standards when thinking about how neorg should do things!

Thanks for all the great work on neorg!

[mrossinek]

Where exactly would this = Bibtex references.bib be in a document?

This syntax is called an insertion in the norg format. It can be used to insert automatically generated text into the document (e.g. = ToC Table of Contents where ToC is the "insertion command" and Table of Conents is an argument which, in this case, becomes the title).
I propose that the insertion = Bibtex <filename(s)> can be used to insert all labels. I prefer this over document.meta because for me, references are not really metadata of the file. Furthermore, the insertion can also be used to mark the place where the bibliography/references section can be inserted into the document.

I would consider retaining the '@' of the labels.

Thanks for the suggestion. This is a very specific implementation detail so for now I think we will just note this here and get back to it once we actually start implementing that.

[jghauser]

Thanks for the explanation!

I do like the idea of using the place where you 'import' the labels as the marker of where the bibliography will go -- it seems nicely minimalist. However, will it also be possible to have citations without having a bibliography? This is quite a common occurrence.

I guess eventually it would be nice to also support file formats other than Bibtex. I don't use these personally, but they do seem to get more and more common.

How do you think the CSL information could be included to define the citation style?

[mrossinek]

However, will it also be possible to have citations without having a bibliography?

Yes ofcourse. For this we could figure out some extension of the footnote syntax:

^ Author2021
Actual reference contents goes here.

Obviously this requires a manual formatting of the reference text.

I guess eventually it would be nice to also support file formats other than Bibtex

This is ofcourse also an extension. I just used = Bibtex as an example but noone will stop us from implementing = Biblatex, = CSL json etc...

How do you think the CSL information could be included to define the citation style?

This seems to be relevant only when converting to another format. So this would be imo a candidate for document.meta or some other way that we will come up with in the future of passing additional arguments to pandoc.
One could also envision this as an argument to the Bibtex insertion but this could complicate multiple file handling 🤔
Only the future will tell, I suppose.

[bdarcus]

See also more recent discussion on citation support for djot.

jgm/djot#32

FWIW, I had a hand in the org-cite development (as well as created CSL), and think it's a very well-designed system (e.g. beyond the syntax). In particular, it's highly modular, so that it's easy to mix and match different "processors" to get customized behavior, fontification, etc.

To compare the two syntaxes, however, pandoc (and it's iteration in djot) and org-cite:

The primary difference is the latter has named arbitrary local citation "styles" and "substyles"; the former fixed symbols. There is a trade-off there, obviously: much greater flexibility (it allows, for example, to support the full range of biblatex commands) but at a slight increase in complexity, and a loss in concision.

Also, org-cite has two-levels of affixes, so you can do this (as you can in biblatex, but not in pandoc):

[cite:see ;@SomeAuthor2021 ch. 3;@SomeAuthor2022 pp. 22-25]

It's debatable how critical that is, but for some people it can be, given formatting engines often rearrange the order of the references within the citations. Edit: looks like John is likely to add this to djot's.

Otherwise, they are the same from a modeling POV.

Here's a good representation of org-cite from the article linked above:

Also, I don't really know the export story for neorg, but it may be feasible to integrate a project like this, say via third party plugin, to provide rich citation and bibliography formatting capabilities:

https://github.com/zepinglee/citeproc-lua

Though pandoc also has similar capabilities, also using CSL.

[vhyrro]

@bdarcus that's awesome! If org-cite/djot have good semantics then we're already 50% of the way there and I'd be happy to adopt something that has already been battle tested.

I'll take a look at the syntax for each one and try out some ideas. Norg by its nature prefers symbols over keywords, but there may be a way to get the best of both worlds here, especially since norg links are pretty darn flexible :p

I'll wait on mrossinek and we'll cook up a preliminary syntax draft. Then I can post it here where it can be discussed further :)

[bdarcus]

Norg by its nature prefers symbols over keywords ...

I do find the pandoc syntax better if you're just typing in the citation, say with autocompleting on the citekeys. It's very expressive, but compact, so really nicely stays out of the way while writing.

With org, it's more verbose syntax is better suited to programmatic insertion, though in my emacs package we support both.

[vhyrro]

The meat of the discussion has since moved to the specifications repository (found here: nvim-neorg/norg-specs#15), so I'll close this issue here in the neorg repository to prevent duplication and I recommend anyone interested in the to-be-devised citation syntax to move there!