User Tools

Site Tools


extensions:teemip-ip-discovery-collector

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
Last revisionBoth sides next revision
extensions:teemip-ip-discovery-collector [2019/10/09 10:14] – [TeemIp IPv4 Discovery] cnaudextensions:teemip-ip-discovery-collector [2023/08/08 15:07] – [Discovery functions] cnaud
Line 1: Line 1:
 +{{ icons8-binoculars-48.png}}
 ====== IP Discovery - Collector ====== ====== IP Discovery - Collector ======
 ---- dataentry summary ---- ---- dataentry summary ----
Line 5: Line 6:
 index_hidden        : no index_hidden        : no
 level_hidden        : 2 level_hidden        : 2
-version             : 0.3.0 +version             : 3.1.1 
-release_dt          : 2019-02-09 +release_dt          : 2023-08-07 
-TeemIp              : 2.3+TeemIp              : 3.
 +iTop                : 3.x
 code                : teemip-ip-discovery-collector code                : teemip-ip-discovery-collector
-state               : pilot+localization        : - 
 +state               : stable
 diffusion_hidden    : TeemIp wiki diffusion_hidden    : TeemIp wiki
 product_hidden      : included (standalone) product_hidden      : included (standalone)
 module-lists_hidden :  module-lists_hidden : 
-keyword_tags        : discovery, ping, lookup, scan+keyword_tags        : discovery, ping, fping, lookup, scan
 dependencies_s      :  dependencies_s      : 
-download_url        : https://wiki.teemip.com/extensions/teemip-ip-discovery-collector-0.3.0-425.zip+download_wiki       [[https://sourceforge.net/projects/teemip/files/teemip%20-%20extensions/IP%20Discovery/IP%20Discovery%20-%20Collector/3.1.1/teemip-ip-discovery-collector-3.1.1-354.zip/download|teemip-ip-discovery-collector-3.1.1-354.zip]] 
 +github_wiki         : [[https://github.com/TeemIp/teemip-ip-discovery-collector|teemip-ip-discovery-collector]] 
 +php-max             : 8.1
 ---- ----
 +
 +
 +
 +
 +
 +
 +
 +
 +
 +
  
  
Line 28: Line 43:
 ===== Revision History ===== ===== Revision History =====
 ^  Version  ^  Release Date  ^  Comments  ^ ^  Version  ^  Release Date  ^  Comments  ^
-|  0.4.0  |  Coming soon | Improves IP lookup and handles DNS views |+|  3.1.1  |  2023-08-07  | Use fping as an option \\ Make sure collection order is respected \\ Complete list of optional attributes | 
 +|  3.1.0  |  2023-07-25  | Includes iTop Data Collector Base 1.3.0 \\ Implement collection plan feature of that collector base \\ Insure compatibility with PHP 8.1 | 
 +|  3.0.1  |  2022-09-21  | Align data model to TeemIp 3.x \\ Report number of discovered IPs at subnet level \\ Allow DHCP ranges to be skipped during discovery  |  
 +|  0.5.0  |  2020-10-20  | Improves reporting - Consider “connection refused” for scans | 
 +|  0.4.0  |  2019-10-11  | Improves IP lookup and handles DNS views |
 |  0.3.0  |  2019-02-09  | Initial production release | |  0.3.0  |  2019-02-09  | Initial production release |
  
Line 39: Line 58:
   * a [[extensions:teemip-ip-discovery-extension|TeemIp extension]], called the IP Discovery Extension, that adapts TeemIp's standard data model to store the discovery outputs and enables communication between a TeemIp instance and the different discovery engines deployed on the network.   * a [[extensions:teemip-ip-discovery-extension|TeemIp extension]], called the IP Discovery Extension, that adapts TeemIp's standard data model to store the discovery outputs and enables communication between a TeemIp instance and the different discovery engines deployed on the network.
  
-{{  ipdiscoveryprinciples.png  }}+{{  ipdiscoveryprinciples3x.png  }}
  
 The IP Discovery Application is a standalone PHP software that extends iTop's collectors principles. It has been built on top of [[https://wiki.openitop.org/doku.php?id=extensions:itop-data-collector-base| iTop data collector base]] extension. TeemIp administrators may deploy one or several IP Discovery Collectors on different servers, including the one that hosts TeemIp. Each of these remote applications connects to TeemIp to retrieve: The IP Discovery Application is a standalone PHP software that extends iTop's collectors principles. It has been built on top of [[https://wiki.openitop.org/doku.php?id=extensions:itop-data-collector-base| iTop data collector base]] extension. TeemIp administrators may deploy one or several IP Discovery Collectors on different servers, including the one that hosts TeemIp. Each of these remote applications connects to TeemIp to retrieve:
Line 48: Line 67:
 At regular interval (defined by the administrator through a CRON command) the remote collector retrieves in TeemIp the list of networks that it has to discover and, for each of them, the list of IPs already registered. It, then, scans the network, computes  the delta between what has been discovered and the situation before the scan, stores the results of its findings locally and pushes these results back to TeemIp through a standard synchronization mechanism: At regular interval (defined by the administrator through a CRON command) the remote collector retrieves in TeemIp the list of networks that it has to discover and, for each of them, the list of IPs already registered. It, then, scans the network, computes  the delta between what has been discovered and the situation before the scan, stores the results of its findings locally and pushes these results back to TeemIp through a standard synchronization mechanism:
   * New discovered IPs are created,   * New discovered IPs are created,
-  * Ping, IP lookup or scan statuses are updated,+  * Ping / fping, IP lookup or scan statuses are updated,
   * A few statics on the discovery are documented within each subnet.   * A few statics on the discovery are documented within each subnet.
  
-<note>+<note tip>
 Should your network be compartmentalized by different firewalls, the different instances will allow you to bypass the restrictions imposed by the firewalls: limited pings or udp / tcp connections, DNS not available… Of course, you may install the collector on the server that hosts TeemIp but this is not necessary. Should your network be compartmentalized by different firewalls, the different instances will allow you to bypass the restrictions imposed by the firewalls: limited pings or udp / tcp connections, DNS not available… Of course, you may install the collector on the server that hosts TeemIp but this is not necessary.
 </note> </note>
Line 57: Line 76:
 ===== Licensing ===== ===== Licensing =====
  
-TeemIp IP Discovery solution is licensed under the terms of the GNU Affero General Public License Version 3 as published by the Free Software Foundation. This gives you legal permission to copy, distribute and/or modify it under certain conditions. Read the ’license.txt’ file in the distribution. TeemIp IP Discovery is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE WARRANTY OF DESIGN, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE.+TeemIp IP Discovery collector is licensed under the terms of the GNU Affero General Public License Version 3 as published by the Free Software Foundation. This gives you legal permission to copy, distribute and/or modify TeemIp IP Discovery collector under certain conditions. Read the ’license.txt’ file in the distribution. TeemIp IP Discovery collector is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE WARRANTY OF DESIGN, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE.
  
 ===== Limitations ===== ===== Limitations =====
Line 65: Line 84:
 ===== Requirements ===== ===== Requirements =====
  
-On the application's side, the server must be configured to execute PHP files (minimum PHP version 5.3.6, but more recent versions are better) and  it must have a web access to the TeemIp server through https.+On the application's side, the server must be configured to execute PHP files (minimum PHP version 7.4) and it must have a web access to the TeemIp server through https.
 ===== Installation ===== ===== Installation =====
  
- +<note important>Do not install this extension in your webserver directories (like apache). otherwise your configuration files (with URLs, credentials) may be accessible from outside</note>
  
 Expand the content of the zip archive in a folder of the server that will run the discovery. Expand the content of the zip archive in a folder of the server that will run the discovery.
Line 86: Line 104:
     <itop_password>admin</itop_password>     <itop_password>admin</itop_password>
   
 +    <!-- Class collection sequence -->
 +    <collectors_launch_sequence type="array">
 + <!-- IPv4 addresses -->
 + <collector>
 +      <name>TeemIpDiscoveryIPv4Collector</name>
 +      <enable>yes</enable>
 +      <rank>1</rank>
 + </collector>
 + <!-- IPv4 subnets -->
 + <collector>
 +      <name>TeemIpDiscoveryIPv4SubnetCollector</name>
 +      <enable>yes</enable>
 +      <rank>2</rank>
 + </collector>
 + <!-- IP applications -->
 + <collector>
 +      <name>TeemIpDiscoveryIPApplicationCollector</name>
 +      <enable>yes</enable>
 +      <rank>3</rank>
 + </collector>
 +    </collectors_launch_sequence>
 +    
     <!-- Synchronization parameters -->     <!-- Synchronization parameters -->
     <contact_to_notify></contact_to_notify>     <contact_to_notify></contact_to_notify>
Line 97: Line 137:
     </json_placeholders>     </json_placeholders>
  
-<!-- IP Addresses Parameters -->+    <!-- IP Addresses Parameters -->
     <ip_default_status>unassigned</ip_default_status>     <ip_default_status>unassigned</ip_default_status>
     <ip_default_view>External</ip_default_view>     <ip_default_view>External</ip_default_view>
          
-<!-- Discovery functions -->+    <!-- Absolute path for ping command - "ping" will be executed with no path if not provided-->
     <ping_absolute_path>/bin/</ping_absolute_path>     <ping_absolute_path>/bin/</ping_absolute_path>
-    <dig_absolute_path>/usr/bin/</dig_absolute_path>+    <!-- Absolute path for fping command - "fping" will be executed with no path if not provided--> 
 +    <fping_absolute_path>/usr/bin/</fping_absolute_path> 
 +    <!-- Absolute path for dig command - "dig" will be executed with no path if not provided--> 
 +    <dig_absolute_path>/usr/bin/</dig_absolute_path></parameters> 
 +     
 +    <!-- fping usage may be disabled here --> 
 +    <fping_enable>yes</fping_enable>
  
-</parameters>+    
 </code> </code>
  
Line 113: Line 159:
 | itop_login | Login (user account) for connecting to TeemIp. Must have admin rights for executing the data synchro. | admin | | itop_login | Login (user account) for connecting to TeemIp. Must have admin rights for executing the data synchro. | admin |
 | itop_password | Password for the iTop account. | admin_pwd | | itop_password | Password for the iTop account. | admin_pwd |
 +| collectors_launch_sequence | List of collectors to run | N/A |
 +| collector | Details for a collector class | N/A |
 +|   name  | Name of the collector class | TeemIpDiscoveryIPv4Collector |
 +|   enable  | Enbale or disable the collect | yes |
 +|   rank  | Order of the collect amongst allother collectors | 1 |  
 | contact_to_notify | The email address of an existing contact in TeemIp to be notified of the results of the synchronization. | john.doe@demo.com | | contact_to_notify | The email address of an existing contact in TeemIp to be notified of the results of the synchronization. | john.doe@demo.com |
 | discovery_application_uuid | String that uniquely identifies the remote the IP Discovery Application in TeemIp. The string is automatically created at creation time of the IP Discovery object| FFF3_60D8_FEE6_520D | | discovery_application_uuid | String that uniquely identifies the remote the IP Discovery Application in TeemIp. The string is automatically created at creation time of the IP Discovery object| FFF3_60D8_FEE6_520D |
Line 120: Line 171:
 | ip_default_view | String that defines the default view to be set on discovered IPs (if relevant) | "" | | ip_default_view | String that defines the default view to be set on discovered IPs (if relevant) | "" |
 | ping_absolute_path | Absolute path of the ping tool | /bin/ | | ping_absolute_path | Absolute path of the ping tool | /bin/ |
 +| fping_absolute_path | Absolute path of the fping tool | /usr/bin/ |
 | dig_absolute_path | Absolute path of the dig tool | /usr/bin/ | | dig_absolute_path | Absolute path of the dig tool | /usr/bin/ |
 +| fping_enable | Enable usage of fping command | yes |
  
  
 <note tip>The [[https://www.itophub.io/wiki/page?id=extensions%3Aitop-data-collector-base|Itop data collector]] documentation describes the configuration process in details</note> <note tip>The [[https://www.itophub.io/wiki/page?id=extensions%3Aitop-data-collector-base|Itop data collector]] documentation describes the configuration process in details</note>
  
-<note warning>Since TeemIp 2.3 or iTop 2.5, the access to the REST web services is now restricted to the users having the profile REST Services User.</note>+<note warning>Since TeemIp 2.3 or iTop 2.5, access to the REST web services is now restricted to the users having the profile REST Services User.</note>
  
 ===== Run the discovery ===== ===== Run the discovery =====
Line 138: Line 191:
  
 ^ Option ^ Meaning ^ default value ^ ^ Option ^ Meaning ^ default value ^
-| <nowiki>--console_log_level=<level></nowiki> | Level of output to the console. From -1 (none) to 9 (debug). | 6 (info) | +| <nowiki>--config_file</nowiki> | Specify the full path to the configuration file. The file ''conf/params.local.xml'' is used by default if this parameter is omitted. | empty | 
-| <nowiki>--collect_only</nowiki> | Run the discovery but do not synchronize the data with TeemIp and stores discovery results in a csv file under the ''data'' directory. | false | +| <nowiki>--console_log_level=<level></nowiki> | Level of ouput to the console. From -1 (none) to 9 (debug). | 6 (info) | 
-| <nowiki>--synchro_only</nowiki> | Synchronizes the data previously collected and stored in the ''data'' directory with TeemIpDoes not run the discovery. | false | +| <nowiki>--eventissue_log_level=<level></nowiki> | Level threshold to create event issues. From -1 (none) to 9 (debug). | 6 (info) | 
-| <nowiki>--configure-only</nowiki>Checks (and updates if necessary) the synchronization data source in TeemIp and exit. Does NOT run the discovery nor the synchronization |+| <nowiki>--collect_only</nowiki> | Run only the data collection, but do not synchronize the data with iTop | false | 
 +| <nowiki>--synchro_only</nowiki> | Synchronizes the data previously collected (stored in the ''data'' directorywith iTopDo not run the collection. | false | 
 +| <nowiki>--configure_only</nowiki>Check (and update if necessary) the synchronization data sources in iTop and exit. Do NOT run the collection or the synchronization |
 | <nowiki>--max_chunk_size=<size></nowiki> | Maximum number of items to process in one pass, for preserving the memory of the system. If there are more items to process, the application will iterate. | 1000 | | <nowiki>--max_chunk_size=<size></nowiki> | Maximum number of items to process in one pass, for preserving the memory of the system. If there are more items to process, the application will iterate. | 1000 |
 +| <nowiki>--help</nowiki> | Usage mode to display exec.php help. | |
  
 Once invoked through the command line, the discovery application will: Once invoked through the command line, the discovery application will:
-  - Connect to TeemIp to create its Synchronization Data Sources (or check their definition if they already exist and update them if needed) +  * Create its collection plan by connecting to TeemIp and retrieve: 
-  - Connect to TeemIp to retrieve: +    * The main discovery parameters to use, 
-    - What main discovery parameters to use, +    The list of subnets to discover and their specific settings (with regard the ping, ip lookup and scan functions), 
-    The list of subnets to discover and their specific settings (with regard the ping, ip lookup and scan functions), +    The list of IPs already configured in TeemIp and their attributes, 
-    The list of IPs already configured in TeemIp and their attributes, +  * Connect to TeemIp to create the related Synchronization Data Sources (or check their definition if they already exist and update them if needed) 
-  Ping and / or look up and / or scan all subnets' IPs, +  * Ping or fping (default) and / or look up and / or scan all subnets' IPs according to plan
-  Upload the collected data into TeemIp, +  Upload the collected data into TeemIp, 
-  Synchronize the collected data with the existing TeemIp IPs, +  Synchronize the collected data with the existing TeemIp IPs, 
-  Update the discovered subnets with their respective discovery durations.+  Update the discovered subnets with their respective discovery durations
 +  * Update statistics of discovery application.
  
 ==== Scheduling ==== ==== Scheduling ====
Line 165: Line 222:
 ==== Discovery functions ==== ==== Discovery functions ====
  
-The IP Discovery application discovers networks through standard network functions: ping, IP lookup and port scanning, each of them being individually activated for the applications. Furthermore, a function that is activated at the discovery application level can be inhibited for a subnet. +The IP Discovery application discovers networks through standard network functions: fping, ping, IP lookup and port scanning, each of them being individually activated for the applications. Furthermore, a function that is activated at the discovery application level can be inhibited for a subnet.  
 + 
 +**//FPing//** 
 + 
 +If it has been installed on the server and if it has not been disabled through the [[extensions:teemip-ip-discovery-collector#configuration|configuration file]], the [[https://fping.org/|fping]] command is used by default to ping IPs. Compared to ping, it provides enhanced performances and drastically reduces the time it takes to ping a network. it is called with the following parameters: 
 + 
 +^ Parameter ^ Defined in ^ Default value ^ 
 +| fping_absolute_path | /conf/params.local.xml | /usr/bin/ | 
 +| Ping timeout (s) | IP Discovery Application in TeemIp | 1 s | 
 + 
 +The command is invoked through the exec PHP function. 
 + 
 +   <fping_absolute_path/>fping -r1 -t <timeout> -ga <subnet_first_ip> <subnet_broadcast_ip - 1> 
 + 
 +For instance: 
 + 
 +   /usr/bin/fping -r1 -t1000 -ga 10.128.0.0 10.128.7.254
  
 **//Ping//** **//Ping//**
  
-This is the standard and well known command based on icmp request. It is called with the following parameters:+This is the standard and well known command based on icmp request. It is called when fping is not installed on the server or when it has been disabled. The following parameters are used:
  
 ^ Parameter ^ Defined in ^ Default value ^ ^ Parameter ^ Defined in ^ Default value ^
-| ping_absolute_path | /conf/params.local.xml | <empty_string> |+| ping_absolute_path | /conf/params.local.xml | /bin/ |
 | Ping timeout (s) | IP Discovery Application in TeemIp | 1 s | | Ping timeout (s) | IP Discovery Application in TeemIp | 1 s |
  
 The command is invoked through the exec PHP function. The command is invoked through the exec PHP function.
  
-   <ping_absolute_path/>ping c1 -w<timeout> <ip_to_ping>+   <ping_absolute_path/>ping -c 1 -W <timeout> <ip_to_ping>
  
 For instance: For instance:
  
-   /bin/ping -c1 -w2 10.11.12.13+   /bin/ping -c 1 -W 2 10.11.12.13
  
  
Line 189: Line 262:
  
 ^ Parameter ^ Defined in ^ Default value ^ ^ Parameter ^ Defined in ^ Default value ^
-| dig_absolute_path | /conf/params.local.xml | <empty_string> |+| dig_absolute_path | /conf/params.local.xml | /usr/bin/ |
 | DNS server #1 |IP Discovery Application in TeemIp - Can be a FQDN or an IP | <empty_string> | | DNS server #1 |IP Discovery Application in TeemIp - Can be a FQDN or an IP | <empty_string> |
 | DNS server #1 |IP Discovery Application in TeemIp - Can be a FQDN or an IP | <empty_string> | | DNS server #1 |IP Discovery Application in TeemIp - Can be a FQDN or an IP | <empty_string> |
Line 227: Line 300:
     * If no answered is received, a test is done with TCP.     * If no answered is received, a test is done with TCP.
  
-===== Synchronize TeemIp IPs and subnets =====+According to the parameter //Consider “connection refused” as valid//, a scan that fails with error 111 (connection is refused) may be considered as valid from a discovery standpoint. 
 + 
 +===== Synchronize discovery applications, IPs and subnets =====
 Discovery results of an IP Discovery Application are fed back to TeemIp through standard synchro data sources. Every IP discovery application will automatically create its own synchro data sources and will use them to push its finding and statistics into TeemIp, systematically, at the end of each discovery process. Discovery results of an IP Discovery Application are fed back to TeemIp through standard synchro data sources. Every IP discovery application will automatically create its own synchro data sources and will use them to push its finding and statistics into TeemIp, systematically, at the end of each discovery process.
  
-These synchro data sources can be listed and managed from the Synchronization Data Sources menu under the Admin tools section.+These synchro data sources can be listed and managed from the Synchronization Data Sources menu under the Admin tools section. As usual with synchro data sources, the Status tab will provide information on every synchronization that occurred between the remote IP Discovery application and TeemIp. 
 + 
 + 
 +==== TeemIp Discovery Application ==== 
 + 
 +{{ details_synchrodatasource_applicationdiscovery3x.png }} 
 + 
 +Default parameters for a given data source are defined in the remote IP Discovery application configuration file. 
 +  * The UUID of the application is appended to the name of the synchro data source for an easy identification, 
 +  * Statistics on discovery durations are updated. 
 + 
 +Reconciliation of the Application is made on the UUID. The following attributes may be affected by this synchro: 
 + 
 +^  Name  ^  When ?  ^ Comment ^ 
 +| Last discovery date | Creation and update | | 
 +| Duration | Creation and update | | 
  
 ==== TeemIp IPv4 Discovery ==== ==== TeemIp IPv4 Discovery ====
  
-{{ details_synchrodatasource_ipdiscovery.png }}+{{ details_synchrodatasource_ipdiscovery3x.png }}
  
 Default parameters for a given data source are defined in the remote IP Discovery application configuration file. Default parameters for a given data source are defined in the remote IP Discovery application configuration file.
Line 257: Line 348:
  
 <note tip> <note tip>
-The discovery collector automatically detects if the Zone management extension is installed on the remote application:+The discovery collector automatically detects if the Zone Management extension is installed on the remote application:
   * If this is the case: the collector handles the DNS view attribute brought by the extension.   * If this is the case: the collector handles the DNS view attribute brought by the extension.
-  * If this is NOT the ase: the DNS view attribute is not managed.+  * If this is NOT the case: the DNS view attribute is not managed.
 </note> </note>
 ==== TeemIp IPv4 Subnet Discovery ==== ==== TeemIp IPv4 Subnet Discovery ====
  
-{{ details_synchrodatasource_subnetdiscovery.png }}+{{ details_synchrodatasource_subnetdiscovery3x.png }}
  
 Default parameters for a given data source are defined in the remote IP Discovery application configuration file. Default parameters for a given data source are defined in the remote IP Discovery application configuration file.
Line 274: Line 365:
 | Last discovery date | Update | | Last discovery date | Update |
 | Ping duration | Update | | Ping duration | Update |
 +| Ping # | Update |
 | IP lookup duration | Update | | IP lookup duration | Update |
 +| IP Lookup # | Update |
 | Scan duration | Update | | Scan duration | Update |
- +| Scan # | Update |
- +
-As usual with synchro data sources, the Status tab will provide information on every synchronization that occurred between the remote IP Discovery application and TeemIp. +
- +