Mobile Units reverting back to NOC

All,

I need some help with an MSP 2.5 issue. I have a customer who was running a trial with MSP 2.0.2 and successfully used it to manage devices at a number of sites. They have now purchased MSP and we went through the process this week of upgrading the box to 2.5 and rebuilding their setup.

We have added the sites that managed hardware has been deployed to and wiped the old MSA from the mobile units. We have brought their WS5100s up onto the system and moved them from the Unknown site to their designated sites however we are having an issue with the mobile units (MC9060-G)

When we rapidly deploy them they appear in the Network Operations Centre, we then move them to their appropriate site, however next time they check in they go straight back to the NOC.

I could be missign something really obvious here but I can't work out what is going on. Anyone got any ideas?

HELP!!

Steffen Pohl
It seems my customers are not

It seems my customers are not the only ones who have these kind of problems. Would it be possible in the future to provide a bit more of background information in the MSP user guide on how a certain functionality is working.
The "Move to Site" functionality suggests the customer can move their MU's to any site they have defined w/o any restrictions. The reality is different and our customers are not happy with the work-arounds we are currently discussing.
Assuming most of our customers have L3 agents and they do not need the L2 WNMS feature, which is responsible to provide info back to MSP, where the MU belongs to.
Would it be possible to change the permissions of the /opt/msamaint/ftpadmin/WNMS directory (chmod 000 ...) to avoid that the MU's are able to upload their info to MSP? This is probably an easier solution instead of creating separate packages and to deploy them to modify the AB settings in the MU.
I have not tested this and I do not know what the impact could be, but what does MSD think about this proposal?

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Allan Herrod
Actually, very little is

Actually, very little is changed, except for consistency, between MSP 2.0.2 and MSP 2.5.
I have customers reporting the exact same issue (device moving to NOC) in MSP 2.0.2.

In MSP 2.0.2, the device would sometimes move based on FTP, but not always.
In MSP 2.5, we made it happen consistently, so at least we KNOW it will always happen the same way and can deal with it.
The only other change, which makes this more noticable, is that in MSP 2.5, L2 periodic check-in is enabled by default.
In MSP 2.0.2, unless someone manually reconfigured the registry, the L2 agent would not do periodic check-in.
Also, in MSP 2.0.2, the L2 agent was optional but is required in MSP 2.5.
So, in MSP 2.0.2, the motion based on FTP might only happen on reboots, and then not reliably.

We did NOT remove the site motion based on WID association, but that did not always work in MSP 2.0.2 either.
I have many complaints of it just not happening, even when it was apparently set correctly.
In MSP 2.5, it is not yet clear to me if the site motion based on WID works any more reliably than it did in MSP 2.0.2.
I have no evidence yet that it works any less reliably either.
However, the fact that FTP now reliably moves devices may override motion based on WID association.

We have NEVER had site motion based on IP address.
Site motion based on WID association was and is based on the BSSID reported by the device and the MU table reported by the WID.
This is done infrequently because it is a time consuming operation as it has to correlate the WID association of ALL devices.
The default interval for this check is every 24 hours although it can be set as low as every 1 hour.
The FTP motion check is much less costly, since the device originates the WNMS file.
But I get why there are times you DON'T want it.

As for staging at one site and use at another site, the fact is that this was NEVER a planned use case for MSP.
MSP RD was ALWAYS envisioned as staging at the point of use.
In retrospect, this is a shortcoming, and we plan to fix it in MSP Stage where we address this case directly.
In MSP 2.5 and even MSP 2.8, however, there is no good way to address this use case.
When the staging network and the production network differ, you are in the realm of special solutions and workarounds.
For example, after staging a device, you can scan another RD profile to change it to the production network.
This can change the WLAN settings but does not change the FTP settings unless you actually provision a package.
Provisioning a package over the production network may not be possible at the staging site.
In order to make this work, it would be necessary to scan the second profile at the production site.
We will correct that in MSP stage, but can't do much in the short term.

I would be concerned about changing the permissions of the folder on the FTP server.  It might work, but who knows.
If what you want is to disable site motion based on the L2 agent, there is another option.
You can turn off the WNMS flag on the AirBEAM Setttings (using the method described in my prior post).
This can be done with ONE standard package that is pushed to all devices.
Using L3 provisioning, this does NOT depend on where the FTP sever of AirBEAM is pointing.
This will allow the device to check in with whatever FTP it is pointing to, but not upload the WNMS file.
I think its better to take the bullets out of the gun rather than wear a bullet-proof vest.

With regards to using multiple sites on the MSP server using path, keep in mind that this was an afterthought.
It was never part of the original design and was something that was thought up to try and make sites more useful.
It has significant flaws because of the need to duplicate packages to make it work properly.
Sites in MSP, unfortunately, were designed around the idea of FTP servers and are tied to them quite heavily.
In retrospect, that may not have been such a good idea, but it is what was designed.
We will try to lessen that reliance in MSP stage.

There is a LOT of information about how all this stuff works in the Deployment Scenarios Guide.
If there are specific things not covered, please let me know and I will try and cover them better in the MSP 2.8 version.
In particular, I will try to address the shortcomings of the staging vs. production network issue and discuss workarounds.
If you have specific recommendations for things you would like to see, please suggest them.

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Allan Herrod
There is no known specific

There is no known specific limitation on groups.
One thing to remember is that MSP automatically creates one or more groups for every site.
So, if you already had that many sites, then you already had at least that many groups.
There should be no fundamental difference between user-created and automatic groups from a performance standpoint.
But, keep in mind that there is no hierarchical support for groups (unlike for sites), so the groups will all be in a linear scrolling list not a tree.

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Allan Herrod
Damian is correct.The moved

Damian is correct.
The moved devices report will only show devices that were moved from a known site to another known site as a result of WID association or FTP WNMS file extraction.
Using groups instead of sites will mean that the moved devices report will not provide value.

If you are using something like dynamic groups based on IP address, the membership of the group can be used tell which devices currently meet the group criteria.
You can also use the "Device added to group" to detect when a device has changed from one group to another.
You can also use events such as IP Address Change to indicate that devices have moved.
This may or may not be adequate, but it's worth a look.

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Allan Herrod
Ian; This sounds like

Ian;

This sounds like expected behavior given what you have described as your process to this point.

MSP assigns a device to a site based on a WNMS file uploaded by the L2 agent to an FTP server.
When a device is staged using RD via the NOC, the WNMS file is uploaded to the NOC FTP server.
This causes the device to be assigned to the NOC site.
The L2 agent is left pointing to the NOC FTP server as well.
Each time the L2 agent checks in, it uploads another WNMS file to the NOC FTP server.
Each time MSP finds a WNMS file from a device on the NOC FTP server, it moves the device to the NOC site.
The L2 agent does this every 15 minutes, by default.
If you manually move the device to another site, it will get moved back to NOC.
This is normal behavior.

There are a few ways to change this.

  1. Stage the device from the site at which it will be used, instead of from the NOC site.
  2. If the device MUST be staged from the NOC site, then stage it again from the site at which it will be used.
    This second staging might or might not need to change the WLAN settings.
    This second staging SHOULD change the FTP server to the right site and pull SOME package (such as discover).
    This will cause future L2 check-ins to happen to the proper site.
  3. You can capture the AirBEAM.reg file from \Application and edit it to change the FTP information to point to the correct site.
    The updated AirBEAM.reg can be put into an AirBEAM package and pushed to the devices at the site.
    This will work since L3 provisioning does NOT rely on the L2 agent FTP server settings.
    You will either need to use abmerge.exe to merge these settings immediately or mark the AirBEAM package to do a warm boot.

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Ian Jobson
Allan, Thanks for the quick

Allan,

Thanks for the quick response, however, I'm not certain that what you are suggesting will solve our issue.

We are not using local FTP servers, the only FTP resource is the MSP box which all sites talk back to. When we RD the devices they are in the store that they are to be used in and, therefore, on the appropriate subnet which is different to the MSP box (for example, MSP is 172.19.60.10, and the data network for the store we were in yesterday was 172.28.51.x). The sites are set up with 172.19.60.10 as both the MSP accessible FTP IP and the Device accessible FTP IP, and we have set the FTP path as "/Store Name".

With the 2.0.2 install we could use the MSP FTP server and move the terminals into their respective sites and they would stay there, I had assumed based on the subnet or by association to the Wireless Switch. If this has changed in 2.5 such that unless a terminal is talking to a local FTP server it will always appear in the NOC then this is a big problem.

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Allan Herrod
Ian; OK, that is some new

Ian;

OK, that is some new information I did not have that you have additional sites on the same FTP server.
Even though you are using the same FTP server, you have different paths on that server for each site.
In MSP, a site is identified by a unique combination of FTP Server IP Address, username, and path.
Sites are treated the same regardless of which differentiating criteria are used to identify the sites.

If the L2 agent is set to point to the NOC site, then it will cause the device to move back to the NOC site.
You can change the L2 agent to point to the proper site, in this case by changing the path the L2 agent uses.
If you change the path, then when the L2 agent checks in, it will put the WNMS file in the proper place for that site.
Then, when MSP picks up the WNMS file, it will see that it was from the location associated with that site.
It will then assign the device to the proper site.

So, everything I said in my prior post still applies.
You can use any of the described means to update the L2 agent to point to the proper site.

As another alternative, you can disable L2 agent check-in to prevent site motion based on FTP.

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Damian Stock
Allan, If this is the only

Allan,

If this is the only way to overcome this problem, is there an easily way to change the L2 client to not check in or to change its path? I don't beleive that this is an easy thing to achieve from central. Do you have to send a package down with a new AB Smart configuration file? I have at least one customer SPR in place at the moment for this problem so anything that helps me resolve it for them would be appreciated.

Thanks,

Damian

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Ian Jobson
All, I did some testing on

All,

I did some testing on this with my customer yesterday with the following results.

1) Setting up dynamic groups based on IP subnet works fine. Its not as pretty as the hierarchical structure of the site "View Network" tree but it does the job. My one question around this method is do we have the functionality to import a list of groups in the same way that we can with sites (i.e. by pulling in a .csv file)? If not then for a large estate it could be a labourious task inputing all of the different groups.

2) By unchecking the "Enable WNMS" flag in step 2 ("Airbeam Package Installation") the RD profile we found that the terminals would actually stay in the "site" that we had moved them to, even after a warm / cold boot. Obviously the terminal will then be using the MSPA to pass back the information which means that we would have to edit the data collection profile to pass the info back more regularly than the default once per day.

We are currently using the "Mobile Device Static" data collection profile, so what I need to understand is, if I disable WNMS, but then decrease the collection interval for my data collection profile from "1 day" to "15 minutes" what kind of impact am I going to put on the network? How much data is being pulled back by the WNMS packet currently every 15 minutes? And therefore what would the overhead be on switching to a pure MSPA based collection?

Obviously once you start scaling this up across a large estate the impact could be far too big to make it viable, however, if it is fairly low and there is potential to reduce it further then this could be a suitable work around.

 

My biggest concern and comment around this is that the customers that are typically going to look at deploying MSP are those customers with a large number of sites, typically retail stores. These are also the very same people that, in my experience, are currently doing their best to reduce the number of servers or equivalent that are being put in each remote location. This, therefore, means that I suspect the typical scenario for MSp is one where there is no remote FTP server at each site and everything is going to go back to the MSP box. If this is the case then most of our MSP sales are going to come up against this very issue and using Groups or compremising on the data collection - either by increase the network traffic or decreasing the granularity of that collection  - will not always be an acceptable option.

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Allan Herrod
Ian; By default, the L2 agent

Ian;

By default, the L2 agent (AirBEAM) is set to upload a WNMS file every 15 minutes, although this can be changed.
The data collected is minimal and fixed.   There is no way to select what will be collected by the L2 agent.
The size of a WNMS file varies based on the number of AirBEAM packages on the device, since it includes a list of those packages.
In most cases, a WNMS file should be less than 1KB in size.

The L3 agent sends XML documents containing monitoring data based on the data collection profiles selected.
The default Mobile Device Static profile is collected once per day and is around 12KB in size.
Other standard profiles (excluding debugging profiles) are around 11KB in size.
The collection frequency of these profiles varies from once per 2 hours to 4 times per hour and can be adjusted.
Debugging profiles are 30KB or more in size, typically collected much more frequently from only one or a small number of devices.
Additional profiles that collect more or less data, more or less frequently, can be defined.

To be useful, most monitoring data really needs to be collected multiple times per day.
When possible, one or more collections per hour would provide useful data for trending and analysis.
For example, 8 samples of battery level across an 8 hour shift will provide a reasonable chart of battery drain.

With the default collections from L2 and L3, you are currently looking at less than 4KB per hour per device.
Disabling L2 data collection (WNMS) and enabling L3 collection 1/hour using a standard profile, would increase to 11KB per hour.
This would represent a 3x increase in traffic, but likely would provide more usable (and selectable) data content.
If necessary, this could be reduced by cutting collection frequency to once per 2 hours and reducing the number of attributes collected.

For completeness, I should also point out another key difference between L2 and L3 monitoring.
L2 monitoring data is sent to the FTP server by the device and then pulled from the FTP Server by MSP.
If there IS a site FTP Server, this allows monitoring data to collect in the FTP Server and be pulled in "batches" by MSP.
If there is no site FTP Server, and MSP is serving as the FTP server, then all traffic is direct to MSP.
L3 monitoring data is ALWAYS sent directly by the device to MSP.

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Allan Herrod
Ian; I am not sure we would

Ian;

I am not sure we would always want the "Move to Site" function of MSP to override the site assigment by FTP or WID association.
To do so would effectively lose the ability to detect "Moved" devices.
But I could see it as an option, where we could instruct MSP to NOT allow devices to move from their assigned site, even if reason exists to think they should be.
This option would help in the cases you are describing.
This could be a useful thing to file a GRIP about.

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Damian Stock
Allan, I think that the WID

Allan,

I think that the WID association should take priority as this gives a true indication of where the device really is (or was last seen), followed by the FTP (if available or for L2 devices at least) and manual move after that.

What I would love is an AirBeam package to give to my customer that turns off the ABS client schedule settings. I've been testing this and it seems to resolve their issue nicely.

Thanks,

Damian

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Ian Jobson
What he said The WID

What he said

The WID association is the key if you know where a switch, access port, or access point is, and your mobile unit is associated to one of those devices then I think its more than fair to assume that the mobile device is in the same location.

The FTP "Site" should then be the secondary route, with a manual "Move" being the final option. And Allen you are correct it should be an option where you can disable the default operation such that MSP doesn't just automagically move things around.

I'll put a GRIP request in an add the number to this forum when I have it.

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Allan Herrod
Damian; If the basic problem

Damian;

If the basic problem is that the AirBEAM settings on the device point to the wrong site, then there are only 2 possible solutions:

  1. Change the AirBEAM Settings to point to the right site so it checks in and gets moved to that site
    (this will cause the device to move to the correct site each time AirBEAM checks in)
  2. Change the AirBEAM Settings so it does not check in on a scheduled interval
    (note that this will NOT prevent AirBEAM from checking in on a warm or cold boot, hence it STILL might be moved to NOC)

Both of the above fixes would require pushing a package with new AirBEAM Settings.
In either case, what I would likely do is NOT try and push a new AirBEAM.reg file, since that would mean capturing the file first.
What I would do is make a small .REG file with just the CHANGES.
Put this file in an AirBEAM package along with the abmerge.exe utility to merge it into the registry.
After this file is merged into the registry, use the saveabreg.exe utility to write all the AirBEAM settings to AirBEAM.REG.
This approach allows changing JUST the things you want and merging them into the existing AirBEAM.REG.
Note that the small .REG file should NOT be placed in \Application since we DON'T want it merged on a subsequent cold boot.
Since its content has already been merged into AirBEAM.REG, it can actually go away once it is merged.
This can be done using the Delete After Install option on the AirBEAM Package Builder.

Of the 2 above approaches, I would prefer to change the device to check in to the proper site.
This would mean that the device would not have to be manually moved to the site, since it will move automatically.
It would also prevent the device from moving back to NOC each time it is rebooted.

Note: if the device were staged from the right site to begin with, it would not need to be changed.
If the device is pointing to the NOC site, I assume it was staged from the NOC site.
The device COULD have been staged from its actual destination site and this would not be necessary.
I suspect it was not staged from the destination site for one of the following reasons:

  1. The sites are all on MSP and this would have required duplication of all the packages to each site
  2. This would have required printing (and keeping separate) one barcode sheet per site
  3. It was not realized that using the "Download" option to print barcodes always points to NOC
    (to get proper barcodes for a site-relative RD profile you must send the profile to the sites and print from there)

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Damian Stock
Allan, The fundamental issue

Allan,

The fundamental issue here is that the MSP 2.5 L3 client site management architecture is flawed, when compared to MSP 2.0.2 and earlier versions. Regardless of where a device is staged, MSP should recognise that a L3 device has moved locations (the assigned vs current site architecture), which is a very realistic scenario in many customer environments, and the MSP system should reflect this move. This should only be based on the new IP address reported by the L3 client, or the association status of the device, not where the L2 component is reporting back through.

If all L3 devices are staged centrally and then shipped to their ultimate destination (store, DC, etc.), as has been the case in all of my customer installations, then MSP needs to recognise that fact when the device re-appears at its new destination and updates its database accordingly. It should have no impact on MSP what the L2 component of a L3 client does as far as this is concerned. If this is not the case then this needs to be fixed, and quick.

Having to modify the L2 component every time a device moves location is unrealistic and not aligned with the original architecture. This should only be necessary on an L2 client and, even then, it does not allow a customer to track the true location of their L2 clients easily or logically.

Thanks,

Damian

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Ian Jobson
Allan, Just to fill in all

Allan,

Just to fill in all of the gaps. Our scenario is as follows;

We have MSP in a central location (172.19.60.10).

This is the only FTP server but we have defined all of our "Sites" with an individual FTP path, e.g. /Gillingham.

The handhelds (MC9060s) are staged using rapid deployment at their final location, i.e. the store they are to be used in, which has its own DHCP server locally on site supplying addresses.

We are using Level 3 agents, but obviously as part of the RD we deploy Airbeam 2.67 to the devices.

The WS5100s are discovered by SEMM and appear in the "Unknown Site" list. These are then manually moved to their appropriate site.

The MC9060s when RD is finished appear in the "NOC" and if we then manually move them to their appropriate site, then the next time they check in they go back to the NOC.

Prior to the MSP upgrade when we were still using 2.0.2, If we had moved the WS5100s to their site then as soon as a handheld came on board with an IP address in the same subnet it would automatically be moved into the correct site. This functionality appears to have been compremised in MSP2.5.

 

From your messages so far, the way I understand it is that the airbeam settings (essentially the Level 2 agent detail) is defining where the terminal should talk back to. This would mean that for each site we would have to either have a separate RD profile, or after we undertake the Rapid Deployment of a terminal we then have to manually adjust the registry settings to force it into the correct MSP site.

This would not be such an issue for just a couple of sites but we are talking about a 350 site roll out so it just doesn't scale.

What would be the impact if we disable WNMS in the RD profile?

I suspect that if this is expected fuctionality and that the only way around it is to have site specific RD profiles or manual registry changes then there will be significant push back from those customers who have already been using 2.0.2.

IJ

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Creighton P. Wh...
Ian have you considered

Ian have you considered creating dynamic groups which will place a terminal or wireless switch into a group based upon its ip address. The group could be named as per a site.

 

regards

 

Pete  

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Ian Jobson
Pete, I hadn't ... but

Pete,

I hadn't ... but looking at it that's not a bad solution. It doesn't look as pretty as the network tree but it could be a work around. Only question I would have is whether there is a limit on the number of groups that you could define, bearing in mind we are looking at a 300+ site deployment.

IJ

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Damian Stock
Pete, The major issue I see

Pete,

The major issue I see with this is that the "Moved Devices" report would still not reflect devices moving between these groups. It only works between sites in the hierarchy (as far as I can tell given that it is broken anyway). If this is not a problem for the customer then setting up these groups is not a bad option and I have suggested this to at least on of my customers who needs this problem fixed.

Thanks,

Damian

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Ian Jobson
Thanks Allan, that's just

Thanks Allan, that's just what I needed.

So basically the dicussion that needs to be had with a customer who does not have individual FTP servers at each of their sites is as follows;

1) Does it matter if the mobile devices appear in the NOC as opposed to the correct site (to which the answer I suspect would generally be yes)

2) Would they be happy to use groups instead of sites. The compremise being that they lose the hierarchical structure available in the Site view and would have to enter each group manually without the ability to import a list.

3) If the group structure is not appropriate then they would have to assess the overhead on the network of disabling the WNMS packets and decreasing the interval between the MSP data collection. With the default collection profile sending back 11kb each time its collected.

It would be nice if we could get the "Move Site" function to overwrite the settings in the WNMS setup. But in the mean time it looks like there are a couple of solutions.

IJ

Vote: 
Vote up!
Vote down!

Points: 0

You voted ‘up’


Log in to post comments