Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UTXO Consolidation Helpers #3645

Open
arboleya opened this issue Jan 30, 2025 · 10 comments · May be fixed by #3717
Open

UTXO Consolidation Helpers #3645

arboleya opened this issue Jan 30, 2025 · 10 comments · May be fixed by #3717
Assignees
Labels
feat Issue is a feature

Comments

@arboleya
Copy link
Member

arboleya commented Jan 30, 2025

Problem

  1. Users may have a lot of dust coins
  2. A transaction can't have more than N inputs
  3. When funding a transaction, it may require more than N coins
  4. Users have no way to consolidate their coins
  5. As a result, they might get stuck

Actionable Items

We need to think about two phases:

  1. Exporting method utilities for UTXO split and consolidation
  2. Provide an automated flow to consolidate coins on the fly during transaction funding/submission
    • This may be done later, expand the collapsible below for more info.

Automatic UTXO Consolidation

Because UTXO consolidation will use gas, it's tricky to enable it by default.

However, we could..

..offer an extra submission method or add parameters to the existing one.

The idea is to have the flow available, but it should be opt-in.

Here's a first sketch:

---
title: Automatic UTXO Consolidation
theme: dark
---

%%{
  init: {
    'theme': 'dark'
  }
}%%

flowchart TB
  Start(Start) --> GetCoins{GetCoins}
  GetCoins --> IsEnough[[Is it enough?]]
  IsEnough --> Yes(Yes)
  IsEnough --> No(No)
  Yes --> Submit{Submit}
  Submit --> End(End)

  No --> HasFreeInputs[[Max Inputs Reached?]]
  HasFreeInputs --> FreeYes(Yes)
  HasFreeInputs --> FreeNo(No)
  FreeNo --> Error(ERROR<br/>Insuf. Balance)

  FreeYes --> AutoConsolidate[[Auto consolidate?]]
  AutoConsolidate --> AutoNo(No)
  AutoConsolidate --> AutoYes(Yes)

  AutoYes --> Transfer{Transfer}
  Transfer --> GetCoins;
  AutoNo --> Error

style Error stroke:#570f43, fill:#570f43, color:#ffffff;

classDef StartEnd stroke:#333, fill:#fff, color:#000;
class Start StartEnd;
class End StartEnd;

classDef Actions stroke:#10375b, fill:#10375b, color:#ffffff;
class Consolidate Actions;
class Submit Actions;
class GetCoins Actions;
class Transfer Actions;
Loading
@arboleya arboleya added the feat Issue is a feature label Jan 30, 2025
@nedsalk nedsalk self-assigned this Jan 30, 2025
@Torres-ssf
Copy link
Contributor

This is a great sketch. As mentioned, enabling this behavior by default can be tricky since it could result in multiple transactions being submitted upfront.

Providing this functionality through additional parameters is a good approach, as it allows users to control when and how consolidation occurs.

@arboleya
Copy link
Member Author

arboleya commented Feb 5, 2025

@nedsalk Let's focus on the helper methods first - did you think about a simple API to get started?

const account = new Account(ADDRESS, provider);
account.consolidateCoins()
account.splitCoin()

@nedsalk
Copy link
Contributor

nedsalk commented Feb 6, 2025

@arboleya below is the full consolidation workflow for both base and non-base assets. The complex solution aims to be smart and take into consideration the "optimal" base+non-base asset combinations. The simple solution consolidates all base assets into one before consolidating any non-base assets.

UTXO consolidation (complex)
flowchart TD
    Start["Consolidate coins"] --> 
    |Select asset id| get_all_coins("Get coins (all pages)") --> 
        more_coins_than_max_inputs(count >= max_inputs?) --> 
            |No| is_base_asset("Consolidating base asset?") -->
                    |No| get_resources_for_less("Call getResourcesToSpend to fund tx with one base input, all non-base inputs, and two outputs") -->
                        fill_up_all("Combine received base asset resources and as many non-base asset coins as possible") -->
                        at_least_two_non_base("There are at least two non-base inputs in the combination?") --> 
                            |Can't consolidate one coin, too many base asset dust coins, consider first consolidating them| throw
                        at_least_two_non_base -->
                            |Yes| verify_resources_can_fund("Verify resources can still fund tx after they have been added") -->
                                |No| refund_tx("Call getResourcesToSpend to fund this updated tx") -->
                                    fill_up_all
                            verify_resources_can_fund -->
                                |Yes| prepare_script_tx("Prepare tx with coins, check validity (amount > maxFee)")
                is_base_asset -->
                    |Yes| select_all_coins("Select all coins") -->
                    prepare_script_tx
        more_coins_than_max_inputs --> 
        |   Yes| is_base_asset_2("Consolidating base asset?") -->
                |No| get_resources_two_outputs("Call getResourcesToSpend to fund tx that has max fake inputs and two outputs") -->
                    fill_up_max("Combine funding resources and consolidation coins until max_inputs reached")
                is_base_asset_2 -->
                |Yes| get_resources_one_output("Call getResourcesToSpend to fund tx that has max fake inputs and one output") --> 
                    fill_up_max("Combine resources for funding and consolidation coins until max_inputs reached") --> 
                    prepare_script_tx
        prepare_script_tx --> 
            |Invalid| throw
        prepare_script_tx -->
            |Valid| send_tx("Send transaction and await")
        send_tx --> |Success| coins_remaining{"Remaining unconsolidated coins > 1?"}
        send_tx --> |Failure| throw("Throw, put results of previous transactions on error")
        send_tx --> |SqueezedOut| throw
        coins_remaining --> |Yes| more_coins_than_max_inputs
        coins_remaining --> |No| return_to_user(Return transactions and coins to user)
Loading
UTXO consolidation (simple)
flowchart TD
    Start(Start) -->
    select_asset[\Select asset id/] -->
    get_all_coins{"get all coins (paginated)"} -->
    has_coins([Has more than one unconsolidated coin?]) ---> |No| return@{shape: docs, label: "Return transactions <br>(and all coins?)<br>to user"}
    has_coins --> |Yes| consolidating_base_asset([Consolidating base asset?])
    consolidating_base_asset -->
    |No| consolidate_base_asset((("Consolidate all base asset UTXOs into one"))) -->
    |Continue| combine_non_base@{shape: procs, label: "Select one base and <br>(max. inputs - 1) non-base coins"} -->
        build_tx((("Build tx and check validity <br>(amount > max fee)")))

    consolidating_base_asset --> 
    |Yes| more_than_max_inputs([coins > max. inputs?]) -->
        |No| select_all@{shape: procs, label: "Select all base coins"} --> build_tx
        more_than_max_inputs -->
        |Yes| get_resources{"Call getResourcesToSpend to fund tx with max inputs"} --> 
            combine_base@{shape: procs, label: "Combine resources and unconsolidated coins until max. inputs"} --> 
            build_tx
    
    build_tx -->
    |Invalid| throw@{shape: docs, label: "Throw, put results of previous transactions on error"}
    build_tx -->
    |Valid| submit{"Submit"}

    submit -.-> |Success| has_coins
    submit --> |Failure| throw
    submit --> |SqueezedOut| throw


style consolidate_base_asset fill:none;
style return stroke:green, fill:none, color:green;
style throw stroke:red, fill:none, color:red;

classDef HTTPReq stroke:blue, fill:none, color:blue;
class get_all_coins HTTPReq;
class get_resources HTTPReq;
class submit HTTPReq;
Loading

@arboleya arboleya changed the title Automatic UTXO Consolidation UTXO Consolidation Helpers Feb 7, 2025
@nedsalk nedsalk linked a pull request Feb 18, 2025 that will close this issue
4 tasks
@nedsalk
Copy link
Contributor

nedsalk commented Mar 17, 2025

Selecting base asset coins in-memory when consolidating non-base assets

The getCoinsToSpend query doesn't return only the base assets that are enough for a consolidation transaction but rather returns more coins than that, presumably as a way of consolidating dust coins for those that use getCoinsToSpend and not select their own UTXOs. This functionality, however, comes into conflict with our non-base asset coin consolidation because they're fighting for the limited max_inputs space. Below is a flowchart detailing how we could select base asset coins in memory to guarantee a higher amount of consolidated non-base asset coins.

Base asset coin selection
Note Description
(1) optimal maxInputs - consolidationCoins.length + 1 base coins of lowest value that satisfy tx fee.
(2) least satisfactory Highest value base coins combination that satisfes tx fee.
(3) select consolidation coins Sorted in ascending order, first consolidationCoins.length - selectedBaseCoins.length coins.

flowchart TD;
    start[Start] --> getOptimal("Get optimal <sup>(1)</sup> base coins")
    getOptimal --> amountGteFee1(["amount >= maxFee"])
    amountGteFee1 --> agf1Yes(((yes)))
    amountGteFee1 --> agf1No(((no)))

    agf1Yes --> selectConsolidationCoins("Select consolidation coins <sup>(3)</sup>")
    selectConsolidationCoins --> return@{ shape: docs, label: "return — base + consolidation coins"}
    agf1No --> getMax("Get least satisfactory <sup>(2)</sup> base coins")
    getMax ----> amountGteFee2(["amount >= maxFee"])
    amountGteFee2 --> agf2Yes(((yes)))
    amountGteFee2 --> agf2No(((no)))

    agf2Yes --> selectConsolidationCoins
    agf2No --> throw([INSUFFICIENT_FUNDS_OR_MAX_COINS])

    getOptimal <-.-> |zooming in| optimalCoinsRoot(((—)))
    getMax <-.-> |zooming in| maxCoinsRoot(((—)))

    %% Optimal coins selection
    optimalCoinsRoot --> sortCoinsAsc(["sort coins (asc)"])
    sortCoinsAsc --> iterateOpt["Iterate"]
    iterateOpt --> addCoinOpt([add coin]) --> reachedOptimalLength(["selectedCoins.length == optimalCount"])
    reachedOptimalLength --> rolYes(((yes)))
    reachedOptimalLength --> rolNo(((no)))
    rolNo --> addCoinOpt

    rolYes --> amountGteFeeOptimal(["amount >= maxFee"])
    amountGteFeeOptimal --> agfoYes(((yes)))
    amountGteFeeOptimal --> agfoNo(((no)))

    agfoYes --> returnOptimal{{return coins}}

    agfoNo --> coinsRemaining([coins remaining?])
    coinsRemaining --> crNo(((no)))
    coinsRemaining --> crYes(((yes)))

    crYes --> shiftCoins([shift selectedCoins]) --> addCoinOpt
    crNo --> returnOptimal

    %% Max coins selection
    maxCoinsRoot --> sortCoinsDesc(["sort coins (desc)"])
    sortCoinsDesc --> iterateMax[Iterate]
    iterateMax --> addCoinMax([add coin]) --> amountGteFeeMax(["amount >= maxFee"])
    amountGteFeeMax --> agfmYes(((yes)))
    amountGteFeeMax --> agfmNo(((no)))

    agfmYes ---> returnMax{{return coins}}

    agfmNo --> reachedMaxLength(["selectedCoins.length == maxInputs - 2"])
    reachedMaxLength --> rmlYes(((yes)))
    reachedMaxLength --> rmlNo(((no)))

    rmlYes --> returnMax
    rmlNo --> addCoinMax

classDef Error stroke:red, fill:none, color:red;
class throw Error;

classDef ReturnCoins stroke:#00d5ff, fill:none, color:#00d5ff;
class return ReturnCoins;
class returnOptimal ReturnCoins;
class returnMax ReturnCoins;
Loading

@arboleya
Copy link
Member Author

arboleya commented Mar 19, 2025

Let's reset the rationale here, get back to simplicity, and try to think of two options.

1. Manually

This approach doesn't depend on any new Fuel Core feature.

  • Uses 4x HTTP requests instead of 2x
---
title: UTXO Consolidation
---


flowchart TB
  start(start) --> coins{coins}
  coins --> greaterThanOne([coins.length > 1])
  greaterThanOne --> yes((yes))
  greaterThanOne --> no((no))
  no --> null@{ shape: flag, label: "return null"}
  yes --> estimate{dryRun}
  estimate --> fund{coinsToSpend}
  fund --> canPay([can cover fee?])
  canPay --> canPayNo((no))
  canPayNo --> error@{ shape: docs, label: "return error"}
  canPay --> canPayYes((yes))
  canPayYes --> mergeCoins{{merge coins}}
  mergeCoins --> submit{submit}
  submit --> failure[\failure\]
  failure --> error
  submit --> success[/success/]
  success --> txs@{ shape: docs, label: "return tx"}

style start stroke: #333333, fill: #ffffff, color: #000000
style null stroke: #cccccc, fill: #333333, color: #cccccc
style txs stroke: #0bebcd, fill: #ffffff, color: #000000

style success stroke: #0bebcd, fill: #0bebcd, color: #000000
style failure stroke: #f500b8, fill: #f500b8, color: #000000
style error stroke: #f500b8, fill: #ffffff, color: #000000

classDef Coins stroke: #00d5ff, fill: #00d5ff, color: #000000
class mergeCoins Coins
classDef HTTPReq stroke: #00ffc3, fill: none, color: #00ffc3
class coins HTTPReq
class estimate HTTPReq
class fund HTTPReq
class submit HTTPReq
Loading

2. AssembleConsolidationTx

This approach requires the implementation of a new GraphQL endpoint for assembling consolidation transactions.

  • Uses 2x HTTP requests instead of 4x
---
title: UTXO Consolidation
---

flowchart TB
  start(start) --> coins{"assemble<br/>consolidation<br/>txs"}
  coins --> greaterThanOne([did we get some consolidation tx back?])
  greaterThanOne --> yes((yes))
  greaterThanOne --> no((no))
  no --> null@{ shape: flag, label: "return null"}
  yes --> submit{submitAll}
  submit --> failure[\failure\]
  failure --> error@{ shape: docs, label: "return txs + errors"}
  submit --> success[/success/]
  success --> txs@{ shape: docs, label: "return txs"}


style start stroke: #333333, fill: #ffffff, color: #000000
style null stroke: #cccccc, fill: #333333, color: #cccccc
style txs stroke: #0bebcd, fill: #ffffff, color: #000000

style success stroke: #0bebcd, fill: #0bebcd, color: #000000
style failure stroke: #f500b8, fill: #f500b8, color: #000000
style error stroke: #f500b8, fill: #ffffff, color: #000000

classDef HTTPReq stroke: #00ffc3, fill: none, color: #00ffc3
class coins HTTPReq
class estimate HTTPReq
class fund HTTPReq
class submit HTTPReq
Loading

API

API for both should be identical.

import { WalletUnlocked } from 'fules'

const wallet = new WalletUnlocked(privateKey, provider);

// Prepare and submit consolidation txs in one go
const { txs, errors } = await wallet.consolidateCoins({ assetId: '0x12345...' });

// Or you can prepare consolidation txs first
const {
  txRequests,
  submitAll,
  beforeCount,
  afterCount,
  totalFeeCost,
} = await wallet.assembleConsolidationTxs({ assetId: '0x12345...' });


// And submits them in parallel when you want
await submitAll();

Spec Draft

Important

Write AssembleConsolidationTx Spec.

@nedsalk
Copy link
Contributor

nedsalk commented Mar 20, 2025

@arboleya great idea on AssembleConsolidationTx. Why don't we make it AssembleConsolidationTxs instead? The node can return a list of transactions that it calculates to be the optimal for consolidating UTXOs and we can submit them sequentially, or perhaps even some in parallel if some transactions are not dependent on the change outputs of the others. The great thing about this is that it'd be just one request and we don't need any getResourcesToSpend nor getCoins calls to calculate it on our side.

Write AssembleConsolidationTx Spec.

What we'd need for this is the following GraphQL API:

type ConsolidationTransactionResult {
  inputs: [Input!]!
  outputs: outputs: [Output!]!
  maxFee: U64!
  gasLimit: U64!
}

type AssembleConsolidationTxResult {
  consolidationTxs: [ConsolidationTransactionResult!]
}


type Query  {
  assembleConsolidationTxs(
    owner: Address!
    assetId: AssetId!
    predicate: HexString
    predicateData: HexString
}

Then, based on this data the query returns, our method on our Account can be this:

const { txRequests, submitAll, beforeCount, afterCount, totalFee } =
  await wallet.assembleConsolidationTxs({ assetId: '0x123' });

Which is enough info for browser wallets to use to create consolidation flows.

@arboleya
Copy link
Member Author

Why don't we make it AssembleConsolidationTxs instead?

@nedsalk It's always easier to deal with 1x than with N(edim). We could work with N, but limits must apply, and the scope of the request must be expanded. Bit by bit, we push the finish line further. The basics are required; everything else is optional.

Can you please specify another change about adding a new utxosNum: Int property (or similar) in the balances endpoint that lets consumers know if a given asset is suitable for consolidation, i.e., by having more than let's say ~150-200 UTXOS.

Please, rely on @Torres-ssf to give the final word on the spec to ensure it is as similar as possible with AssembleTx.

@nedsalk
Copy link
Contributor

nedsalk commented Mar 20, 2025

Can you please specify another change about adding a new utxosNum: Int property (or similar) in the balances endpoint that lets consumers know if a given asset is suitable for consolidation, i.e., by having more than let's say ~150-200 UTXOS.

That change would be made on the Balance type that's returned by the balance query:

type Balance {
  owner: Address!
  amount: U64!
  amountU128: U128!
  assetId: AssetId!
  utxosNum: U64!
}

@nedsalk
Copy link
Contributor

nedsalk commented Mar 24, 2025

TODOs:

  • Prepare two transactions ahead of time
  • Make sure the second one can be submitted successfully and that it's dependent on the UTXOs of the first one

@arboleya
Copy link
Member Author

Moved the last two tasks to:

@arboleya arboleya assigned Torres-ssf and unassigned nedsalk Mar 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feat Issue is a feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants