Skip to content
This repository was archived by the owner on Mar 29, 2023. It is now read-only.

Latest commit

 

History

History
463 lines (235 loc) · 14.4 KB

lazycluster.cluster.hyperopt_cluster.md

File metadata and controls

463 lines (235 loc) · 14.4 KB

module lazycluster.cluster.hyperopt_cluster

Module for conveniently managing a Hyperopt cluster.

class MongoLauncher

Abstract implementation of the MasterLauncher interface used to implement a concrete launch strategy for mongodb instance used in hyperopt.

This class implements the logic for starting a MongoDB instance on localhost. Hence, we simply treat the MongoDB instance as master node.

method __init__

__init__(runtime_group: RuntimeGroup)

Initialization method.

Args:

  • runtime_group: The group where the workers will be started.

property port

The port where the master instance is started on. Will be None if not yet started.

Returns:

  • Optional[int]: The master port.

property process

The process object where the master instance was started in.

Returns:

  • Optional[Popen]: The process object or None if not yet started.

class LocalMongoLauncher

Concrete implementation of the MasterLauncher interface. See its documentation to get a list of the inherited methods and attributes.

This class implements the logic for starting a MongoDB instance on localhost. Hence, we simply treat the MongoDB instance as master node.

method __init__

__init__(runtime_group: RuntimeGroup)

Initialization method.

Args:

  • runtime_group: The group where the workers will be started.

property port

The port where the master instance is started on. Will be None if not yet started.

Returns:

  • Optional[int]: The master port.

property process

The process object where the master instance was started in.

Returns:

  • Optional[Popen]: The process object or None if not yet started.

method cleanup

cleanup() → None

Release all resources.

method get_mongod_start_cmd

get_mongod_start_cmd() → str

Get the shell command for starting mongod as a deamon process.

Returns:

  • str: The shell command.

method get_mongod_stop_cmd

get_mongod_stop_cmd() → str

Get the shell command for stopping the currently running mongod process.

Returns:

  • str: The shell command.

method start

start(
    ports: Union[List[int], int],
    timeout: int = 0,
    debug: bool = False
) → List[int]

Launch a master instance.

Note:

If you create a custom subclass of MasterLauncher which will not start the master instance on localhost then you should pass the debug flag on to execute_task() of the RuntimeGroup or Runtime so that you can benefit from the debug feature of RuntimeTask.execute().

Args:

  • ports: Port where the DB should be started. If a list is given then the first port that is free in the RuntimeGroup will be used. The actual chosen port can be requested via the property port.
  • timeout: Timeout (s) after which an MasterStartError is raised if DB instance not started yet. Defaults to 3 seconds.
  • debug: If True, stdout/stderr from the runtime will be printed to stdout of localhost. If, False then the stdout/stderr will be added to python logger with level debug after each RuntimeTask step. Defaults to False.

Returns:

  • List[int]: In case a port list was given the updated port list will be returned. Otherwise an empty list.

Raises:

  • PortInUseError: If a single port is given and it is not free in the RuntimeGroup.
  • NoPortsLeftError: If a port list was given and none of the ports is actually free in the RuntimeGroup.
  • MasterStartError: If master was not started after the specified timeout.

class RoundRobinLauncher

Concrete WorkerLauncher implementation for launching hyperopt workers in a round robin manner.

See the WorkerLauncher documentation to get a list of the inherited methods and attributes.

method __init__

__init__(runtime_group: RuntimeGroup, dbname: str, poll_interval: float)

Initialization method.

Args:

  • runtime_group: The group where the workers will be started.
  • dbname: The name of the mongodb instance.
  • poll_interval: The poll interval of the hyperopt worker.

Raises.

  • ValueError: In case dbname is empty.

property ports_per_host

Dictionary with the host as key and a port list as value. The list contains all ports where a worker instance is reachable on the respective host.

Returns:

  • Dict[str, List[int]]: The ports per host as a dictionary.

method cleanup

cleanup() → None

Release all resources.

method start

start(
    worker_count: int,
    master_port: int,
    ports: List[int] = None,
    debug: bool = True
) → List[int]

Launches the worker instances in the RuntimeGroup.

Args:

  • worker_count: The number of worker instances to be started in the group.
  • master_port: The port of the master instance.
  • ports: Without use here. Only here because we need to adhere to the interface defined by the WorkerLauncher class.
  • debug: If True, stdout/stderr from the runtime will be printed to stdout of localhost. If, False then the stdout/stderr will be added to python logger with level debug after each RuntimeTask step. Defaults to False.

Returns:

  • List[int]: The updated port list after starting the workers, i.e. the used ones were removed.

class HyperoptCluster

Convenient class for launching a Hyperopt cluster in a RuntimeGroup.

HyperoptCluster inherits from MasterWorkerCluster. See its documentation to get a list of the inherited methods and attributes.

The number of hyperopt workers defaults to the number of Runtimes in the used RuntimeGroup. This number can be adjusted so that more or less workers than available Runtimes can be used. Per default the desired number of workers is started in a round robin way as implemented in RoundRobinLauncher. Consequently, this leads to an equal distribution of hyperopt workers in the RuntimeGroup. You can provide a custom implementation inheriting from the WorkerLauncher class in order to execute a different strategy how workers should be started. The master instance (i.e. the mongoDB) will always be started on localhost as implemented in LocalMasterLauncher. This behavior can also be changed by providing a custom implementation inheriting from the MasterLauncher.

method __init__

__init__(
    runtime_group: RuntimeGroup,
    mongo_launcher: Optional[MongoLauncher] = None,
    worker_launcher: Optional[WorkerLauncher] = None,
    dbpath: Optional[str] = None,
    dbname: str = 'hyperopt',
    worker_poll_intervall: float = 1
)

Initialization method.

Args:

  • runtime_group: The RuntimeGroup contains all Runtimes which can be used for starting the entities.
  • mongo_launcher: Optionally, an instance implementing the MasterLauncher interface can be given, which implements the strategy for launching the master instances in the cluster. If None, then LocalMasterLauncher is used.
  • worker_launcher: Optionally, an instance implementing the WorkerLauncher interface can be given, which implements the strategy for launching the worker instances. If None, then RoundRobinLauncher is used.
  • dbpath: The directory where the db files will be kept. Defaults to a mongodb directory inside the utils.Environment.main_directory.
  • dbname: The name of the database to be used for experiments. See MongoTrials url scheme in hyperopt documentation for more details. Defaults to ´hyperopt´.
  • worker_poll_intervall: The poll interval of the hyperopt worker. Defaults to 0.1.

Raises:

  • PermissionError: If the dbpath does not exsist and could not be created due to lack of permissions.

property dbname

The name of the MongoDB database to be used for experiments.

property master_port

The port where the master instance was started. None, if not yet started.

Returns:

  • Optional[int]: The master port.

property mongo_trial_url

The MongoDB url indicating what mongod process and which database to use.

Note:

The format is the format required by the hyperopt MongoTrials object.

Returns:

  • str: URL string.

property mongo_url

The MongoDB url indicating what mongod process and which database to use.

Note:

The format is mongo://host:port/dbname.

Returns:

  • str: URL string.

property runtime_group

The RuntimeGroup.

Returns:

  • RuntimeGroup: The used group.

method cleanup

cleanup() → None

Release all resources.

method start_master

start_master(
    master_port: Optional[int] = None,
    timeout: int = 3,
    debug: bool = False
) → None

Start the master instance.

Note:

How the master is actually started is determined by the the actual MasterLauncher implementation. Another implementation adhering to the MasterLauncher interface can be provided in the constructor of the cluster class.

Args:

  • master_port: Port of the master instance. Defaults to self.DEFAULT_MASTER_PORT, but another one is chosen if the port is not free within the group. The actual chosen port can be requested via self.master_port.
  • timeout: Timeout (s) after which an MasterStartError is raised if master instance not started yet.
  • debug: If True, stdout/stderr from the runtime will be printed to stdout of localhost. Has no effect for if the master instance is started locally, what default MasterLauncher implementations usually do.

Raises:

  • PortInUseError: If a single port is given and it is not free in the RuntimeGroup.
  • NoPortsLeftError: If there are no free ports left in the port list for instantiating the master.
  • MasterStartError: If master was not started after the specified timeout.

This file was automatically generated via lazydocs.